Data Driven Monitoring
•
2 likes•1,456 views
At Etsy we are big fans of graphing and monitoring all the things. We deploy our main site several times a day and our monitoring provides us with the tight feedback loop we need to make this possible. The same goes for changes in the infrastructure which are deployed in a similar fashion of small and frequent changes. For this to work we have build up monitoring that tracks changes and possible problems in every nook and cranny of the Etsy stack, be it a network change, systems or application level performance or how bad the last week of on-call rotation was. The flipside of monitoring all the things however is that we have a myriad of graphs and alerts that can potentially be important and page the on-call engineer at any given time. The risk of running into alert fatigue and a normalization of deviance through not properly scoped checks is rising with this ever increasing size of the monitoring system. This is why we also continuously monitor our monitoring system and ask questions about whether we have all the information at hand when we get paged, if certain alerts actually need to wake someone up or if they are needed at all. I will give a quick overview of how our monitoring stack is built and then give insights into how we gather data about it and use it to make things better. For site operations and more importantly for the human getting paged when something does go wrong.
Report
Share
Report
Share
Download to read offline
Recommended
Development, Deployment & Collaboration at Etsy
This document discusses development, deployment, and collaboration practices at Etsy. It describes that Etsy deploys changes 50 times per day on average and uses practices like continuous integration, developer VMs, the "Deployinator" tool, monitoring dashboards, on-call rotations, incident response procedures, post-mortem reviews, and knowledge sharing to facilitate frequent deployments in a collaborative environment. The goal is to share practices that work for Etsy and encourage learning and teaching across teams.
Mobile CI at Etsy
Etsy uses GitHub Enterprise for code reviews of code committed to their on-premise git repositories. They have over 350 Linux virtual machines called "The Bobs" that are used to build and test code. Etsy also has 25 Mac Minis that are used to build, test, and run linting on their mobile Android and iOS code using an automated process with Chef. The summary describes Etsy's implementation of continuous integration for mobile apps using GitHub, Linux VMs, Mac Minis, and automation with Chef to improve testing and visibility of their code.
Ansible ALLTHETHINGS
Learn how in less than 6 months and with a 1-person team, they went from no infrastructure automation, to having all of their infrastructure automated with Ansible. Learn how BigPanda (http://bigpanda.io ) handles zero-downtime infrastructure updates and connects Ansible with their chat infrastructure, and some strategies on managing automation projects with very small teams.
Monitoring in the cloud with Puppet
This document summarizes a presentation about monitoring in the cloud using Puppet. It discusses challenges with traditional monitoring like Nagios being difficult to scale in cloud environments. It promotes using Puppet to configure monitoring tools like Collectd, Logstash, Graphite and Icinga in a dynamic and reproducible way. Specific examples are given around using Puppet to export monitoring configurations and collect metrics that are then graphed in Graphite.
Data + Visualization Toronto Meetup 4: The Google Maps API
Trimmed version of slides shown at the Data + Visualization Toronto Meetup Group's Fourth Meetup. Contains information about the Google Maps API, and Google Geocoding API.
Concourse webhook
This document discusses using webhooks in Concourse pipelines to reduce API calls. It begins with an introduction and background on the author. It then provides an overview of Concourse architecture and the roles of the ATC, workers, and resources/jobs. Next, it explains how webhooks can be used resource-agnostically in Concourse by defining a webhook token. It describes how webhooks allow external services like GitHub to notify Concourse of changes instead of Concourse constantly checking. Finally, it mentions a demo, online resources, documentation, Slack community, and invites questions.
ConcourseCI love Minio
Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.
Concourse and Database
- Concourse is a open source CI/CD tool that allows building and testing projects in containers. It uses a pluggable resource interface to check for changes and run builds in isolated containers.
- While Concourse natively supports running builds in containers, it does not support running external services like databases that may be needed to test against.
- One solution is to use "Docker in Docker in Garden" (DCINDG), which runs a Docker daemon inside a Docker container managed by Concourse's Garden container runtime. This allows testing tasks to use Docker Compose to stand up external services in isolated containers.
- The presentation demonstrated this approach with a sample project on GitHub and discussed Concourse concepts like jobs, resources
Recommended
Development, Deployment & Collaboration at Etsy
This document discusses development, deployment, and collaboration practices at Etsy. It describes that Etsy deploys changes 50 times per day on average and uses practices like continuous integration, developer VMs, the "Deployinator" tool, monitoring dashboards, on-call rotations, incident response procedures, post-mortem reviews, and knowledge sharing to facilitate frequent deployments in a collaborative environment. The goal is to share practices that work for Etsy and encourage learning and teaching across teams.
Mobile CI at Etsy
Etsy uses GitHub Enterprise for code reviews of code committed to their on-premise git repositories. They have over 350 Linux virtual machines called "The Bobs" that are used to build and test code. Etsy also has 25 Mac Minis that are used to build, test, and run linting on their mobile Android and iOS code using an automated process with Chef. The summary describes Etsy's implementation of continuous integration for mobile apps using GitHub, Linux VMs, Mac Minis, and automation with Chef to improve testing and visibility of their code.
Ansible ALLTHETHINGS
Learn how in less than 6 months and with a 1-person team, they went from no infrastructure automation, to having all of their infrastructure automated with Ansible. Learn how BigPanda (http://bigpanda.io ) handles zero-downtime infrastructure updates and connects Ansible with their chat infrastructure, and some strategies on managing automation projects with very small teams.
Monitoring in the cloud with Puppet
This document summarizes a presentation about monitoring in the cloud using Puppet. It discusses challenges with traditional monitoring like Nagios being difficult to scale in cloud environments. It promotes using Puppet to configure monitoring tools like Collectd, Logstash, Graphite and Icinga in a dynamic and reproducible way. Specific examples are given around using Puppet to export monitoring configurations and collect metrics that are then graphed in Graphite.
Data + Visualization Toronto Meetup 4: The Google Maps API
Trimmed version of slides shown at the Data + Visualization Toronto Meetup Group's Fourth Meetup. Contains information about the Google Maps API, and Google Geocoding API.
Concourse webhook
This document discusses using webhooks in Concourse pipelines to reduce API calls. It begins with an introduction and background on the author. It then provides an overview of Concourse architecture and the roles of the ATC, workers, and resources/jobs. Next, it explains how webhooks can be used resource-agnostically in Concourse by defining a webhook token. It describes how webhooks allow external services like GitHub to notify Concourse of changes instead of Concourse constantly checking. Finally, it mentions a demo, online resources, documentation, Slack community, and invites questions.
ConcourseCI love Minio
Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.
Concourse and Database
- Concourse is a open source CI/CD tool that allows building and testing projects in containers. It uses a pluggable resource interface to check for changes and run builds in isolated containers.
- While Concourse natively supports running builds in containers, it does not support running external services like databases that may be needed to test against.
- One solution is to use "Docker in Docker in Garden" (DCINDG), which runs a Docker daemon inside a Docker container managed by Concourse's Garden container runtime. This allows testing tasks to use Docker Compose to stand up external services in isolated containers.
- The presentation demonstrated this approach with a sample project on GitHub and discussed Concourse concepts like jobs, resources
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
This document discusses using Chef to automate server maintenance including patching and rebooting. It outlines:
- Current pain points with patching including unclear responsibilities and risk of simultaneous reboots.
- Proposed solution of using Chef to schedule patching and reboots on a periodic basis (e.g. weekends), with options to customize the schedule and opt out.
- Recommendations for testing the Chef cookbook gradually and allowing time for customization before full rollout.
1 year with Docker
This document summarizes John Fiedler's presentation on his company's one year experience using Docker in production. It outlines their timeline of Docker use cases from 2013-2014 including for development environments, continuous deployment, and zero downtime deployments. It describes where they do and do not use Docker for persistent storage, middleware, tools/scripts/jobs, and web/monitoring/logging applications. It also details the operational standards they use like updating, monitoring, logging and security. Finally, it notes issues they encountered with device mapper, private registries, security concerns, and container types.
PuppetConf 2017: No Server Left Behind - Miguel Di Ciurcio Filho, Instruct
A big percentage of companies do not use proper configuration management nor automation. In this talk, I will share various ideas and experiences, making the audience feel much more prepared and confident in rolling out Puppet to manage old or legacy environments that might be left out of automation and configuration management. It is possible to roll out the Puppet agent with ease on these challenging scenarios, letting Puppet gradually fully manage everything with confidence and close to no impact. One of the reasons that commonly makes companies afraid of rolling out the Puppet agent is the impact of actually letting it do its job, since there might be so much accumulated technical debt due to years of manual work. I’ll show successful strategies used on some of our clients that demonstrate Puppet’s powerful built-in state modeling and simulation mechanisms and various useful resource types that let us very granularly manage configuration without breaking what is already in place.
Serverless preview environments to the rescue
End to end testing every commit is a challenge solved with serverless preview environments. Traditionally only select environments like "integration" and "UAT" could be fully tested, because apps required deployment tooling. Using modern serverless cloud solutions, you can deploy and host every commit, providing immediate PR feedback on feature branches and discovering problems long before they're merged.
PuppetConf 2017: Using Puppet Enterprise APIs with PowerShell- Jason Johnson,...
Puppet Enterprise has great APIs that can simplify many aspects of the care and feeding of your infrastructure. While users have had success using Puppet Enterprise API in combination with curl, python and bash shell scripts, implementing these tools on Windows can be problematic. We will discuss PowerShell: what it is and how it differs from other shells; its increasing ubiquity; and tools that simplify coding. We will explore how to convert a curl command to PowerShell for (1) checking puppet status endpoints, (2) a puppet db query, and (3) a code manager deployment. Security is not left out as we discuss the issues with untrusted CAs and creating a pfx certificate to ensure secure commands from PowerShell. With this new knowledge you will be able to evolve the existing curl examples into PowerShell scripts to which your Windows colleagues can not only run, but also contribute.
Configuring SQL Server Reporting Services for ASP.NET Running on Azure Web Role
This document discusses configuring SQL Server Reporting Services for an ASP.NET application running on an Azure web role. It covers creating a database on Azure Database, configuring SQL Server 2012 Reporting Services on an Azure VM, creating an ASP.NET report viewer, and deploying the ASP.NET application to an Azure web role. The presenter demonstrates each step and takes questions at the end.
Stripe con 2021 UI stack
The document summarizes the history and current state of the Silverstripe CMS user interface stack. It describes how the stack originally used a mismatch of legacy technologies like jQuery, Entwine, and templates. It then overhauled to use modern technologies like Webpack, React, Redux, and GraphQL. However, the legacy stack is still in use alongside the modern one, resulting in a hybrid "weird" stack. The document advocates embracing this weirdness and focuses on incremental improvements within these constraints.
React for .net developers
This document introduces React for .NET developers. It provides a brief history of web development in .NET and discusses how React addresses the needs of modern users and developers. The core concepts of React like components, JSX, top-down data flow, and the virtual DOM are explained. ReactJS.NET is introduced as a way to use React with ASP.NET MVC. In summary, React provides fast load times, rich client functionality, and testable code while having a short learning curve and enabling reusability.
From No Git to 3000 GitHub Users and How to Keep Them Happy - GitHub Universe...
A story and lessons we learned about how we went from not using Git, to not only introducing it but making it the default version control system along with GitHub Enterprise for 3000 users at Target.
Modern infrastructure as code with ansible cake fest 2021
This document discusses using Ansible to implement infrastructure as code. It provides an overview of Ansible concepts like modules, inventory, and playbooks. It then demonstrates how to create an Ansible playbook to install and configure PHP on a server. It also discusses best practices like separating environments, using variables, reusing roles from Ansible Galaxy, and secrets management.
Why you should add React to your Rails application now!
React is a javascript library for building user interfaces which has many great features going for it. It also happens to be easy to introduce in small increments, particularly when compared to other javascript frameworks, and plays well with Rails. Find out how you can start modernizing the javascript in your Rails application today, without major refactoring.
Expressjs from-zero-to-hero
This document summarizes Express, a web application framework for Node.js. It notes that Express is minimalist, unopinionated, and fast, processing around 21k requests per second. It discusses key Express features like middleware for request/response processing, error handling, routing, and template engines. It provides examples of middleware, security, and input validation packages that can be used with Express. The overall document introduces Express and its core capabilities for building web applications and APIs.
It Sounded Good on Paper - Lessons Learned with Puppet
This talk is a 12 point guide on the things we did wrong during our journey with Puppet. We hope sharing helps people prevent the same mistakes in the future.
PuppetConf 2016: Puppet & Azure – Kenaz Kwa, Puppet
Here are the slides from Kenaz Kwa's PuppetConf 2016 presentation called Puppet & Azure. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
Sample From Ramesh
This document discusses the basics of using Git and GitHub for source control. It explains some key concepts like branches, committing changes locally, checking for updates from others, and pushing commits to a remote repository to share changes. The document walks through common Git commands like cloning a repository, checking out branches, committing, merging, and pushing to share code.
React, London JS Meetup, 11 Aug 2015
React is a revolution in web UI technology that uses a virtual DOM for fast rendering. It allows developers to build user interfaces as components in a declarative way using JavaScript or JSX. Components are fast, stateless, and their UI is a projection of data. The virtual DOM diffing algorithm efficiently detects changes and updates the real DOM. React promotes building user interfaces with unidirectional data flow and separating concerns into components rather than MVC.
The Ruby workflow
The document discusses a Ruby workflow that focuses on continuous integration and deployment. It is presented by Clemens Helm, a Codeship employee, and proposes a workflow involving developing on feature branches, creating pull requests to merge code to the master branch, and then automatically deploying to production after the code is merged. The workflow aims to get ideas to users quickly through short development cycles, encourage contributions, and ensure code quality while making deployments effortless. However, it notes that the pull request cycle can be a potential downside.
Build Nodejs APIs using Serverless
Serverless lets you focus on coding and testing instead of provisioning infrastructure, configuring web servers, debugging your configuration, managing security settings, and all the drudgery normally associated with getting an app up and running. In this session with, you’ll discover how to migrate an API of an existing app to Azure Functions. You’ll learn how to use Visual Studio Code and the Azure Functions extension to speed up your work. After this session, you’ll join the ranks of serverless developers.
Chef vs. Puppet in the Cloud: How Telepictures and MoneySuperMarket Do It
Doug Babst from Telepictures discussed how they migrated their infrastructure from using RightScript to manage servers in AWS to using Chef. They now define their infrastructure as code using Chef recipes which are version controlled and tested locally using Vagrant before being deployed to AWS. This allows them to more easily and consistently configure their 150 servers.
Jim Davies from MoneySupermarket discussed how they use Puppet to manage their infrastructure in a dynamic cloud environment. They take a masterless approach where instances pull their own configuration from a Git repository. They leverage tools like Facter, Hiera and Librarian-Puppet to parameterize configurations and deploy roles to instances. This infrastructure as code approach allows them to easily deploy their business services across different
IoT Google Cloud Functions with Firebase
Presentation slides used when introducting Google Cloud Functions, as well as Cloud Functions for Firebase, with Internet of Things
Removing Barriers to Going Fast
The document discusses strategies for removing barriers to speed and continuous deployment at Etsy. It describes establishing a culture of open communication, trust, and removing fear of failure in order to optimize for minimizing time to recovery from incidents rather than minimizing incidents. Key techniques discussed include enabling anyone to deploy code, conducting blameless post-mortems, and using automation tools like Deployinator to streamline deployments.
Mean Time to Sleep: Quantifying the On-Call Experience
Starting an on-call rotation can be like opening a door into the unknown. You don’t know if it will be a bad week or if it will be an especially bad week. You don’t know what to expect. Thinking that historical information from past on-call rotations might yield useful insights, Etsy’s Operations team set out to quantify the on-call experience, identify what made it difficult, and use those data to reduce the incidence of pain points in an attempt to make being on call more bearable.
More Related Content
What's hot
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
This document discusses using Chef to automate server maintenance including patching and rebooting. It outlines:
- Current pain points with patching including unclear responsibilities and risk of simultaneous reboots.
- Proposed solution of using Chef to schedule patching and reboots on a periodic basis (e.g. weekends), with options to customize the schedule and opt out.
- Recommendations for testing the Chef cookbook gradually and allowing time for customization before full rollout.
1 year with Docker
This document summarizes John Fiedler's presentation on his company's one year experience using Docker in production. It outlines their timeline of Docker use cases from 2013-2014 including for development environments, continuous deployment, and zero downtime deployments. It describes where they do and do not use Docker for persistent storage, middleware, tools/scripts/jobs, and web/monitoring/logging applications. It also details the operational standards they use like updating, monitoring, logging and security. Finally, it notes issues they encountered with device mapper, private registries, security concerns, and container types.
PuppetConf 2017: No Server Left Behind - Miguel Di Ciurcio Filho, Instruct
A big percentage of companies do not use proper configuration management nor automation. In this talk, I will share various ideas and experiences, making the audience feel much more prepared and confident in rolling out Puppet to manage old or legacy environments that might be left out of automation and configuration management. It is possible to roll out the Puppet agent with ease on these challenging scenarios, letting Puppet gradually fully manage everything with confidence and close to no impact. One of the reasons that commonly makes companies afraid of rolling out the Puppet agent is the impact of actually letting it do its job, since there might be so much accumulated technical debt due to years of manual work. I’ll show successful strategies used on some of our clients that demonstrate Puppet’s powerful built-in state modeling and simulation mechanisms and various useful resource types that let us very granularly manage configuration without breaking what is already in place.
Serverless preview environments to the rescue
End to end testing every commit is a challenge solved with serverless preview environments. Traditionally only select environments like "integration" and "UAT" could be fully tested, because apps required deployment tooling. Using modern serverless cloud solutions, you can deploy and host every commit, providing immediate PR feedback on feature branches and discovering problems long before they're merged.
PuppetConf 2017: Using Puppet Enterprise APIs with PowerShell- Jason Johnson,...
Puppet Enterprise has great APIs that can simplify many aspects of the care and feeding of your infrastructure. While users have had success using Puppet Enterprise API in combination with curl, python and bash shell scripts, implementing these tools on Windows can be problematic. We will discuss PowerShell: what it is and how it differs from other shells; its increasing ubiquity; and tools that simplify coding. We will explore how to convert a curl command to PowerShell for (1) checking puppet status endpoints, (2) a puppet db query, and (3) a code manager deployment. Security is not left out as we discuss the issues with untrusted CAs and creating a pfx certificate to ensure secure commands from PowerShell. With this new knowledge you will be able to evolve the existing curl examples into PowerShell scripts to which your Windows colleagues can not only run, but also contribute.
Configuring SQL Server Reporting Services for ASP.NET Running on Azure Web Role
This document discusses configuring SQL Server Reporting Services for an ASP.NET application running on an Azure web role. It covers creating a database on Azure Database, configuring SQL Server 2012 Reporting Services on an Azure VM, creating an ASP.NET report viewer, and deploying the ASP.NET application to an Azure web role. The presenter demonstrates each step and takes questions at the end.
Stripe con 2021 UI stack
The document summarizes the history and current state of the Silverstripe CMS user interface stack. It describes how the stack originally used a mismatch of legacy technologies like jQuery, Entwine, and templates. It then overhauled to use modern technologies like Webpack, React, Redux, and GraphQL. However, the legacy stack is still in use alongside the modern one, resulting in a hybrid "weird" stack. The document advocates embracing this weirdness and focuses on incremental improvements within these constraints.
React for .net developers
This document introduces React for .NET developers. It provides a brief history of web development in .NET and discusses how React addresses the needs of modern users and developers. The core concepts of React like components, JSX, top-down data flow, and the virtual DOM are explained. ReactJS.NET is introduced as a way to use React with ASP.NET MVC. In summary, React provides fast load times, rich client functionality, and testable code while having a short learning curve and enabling reusability.
From No Git to 3000 GitHub Users and How to Keep Them Happy - GitHub Universe...
A story and lessons we learned about how we went from not using Git, to not only introducing it but making it the default version control system along with GitHub Enterprise for 3000 users at Target.
Modern infrastructure as code with ansible cake fest 2021
This document discusses using Ansible to implement infrastructure as code. It provides an overview of Ansible concepts like modules, inventory, and playbooks. It then demonstrates how to create an Ansible playbook to install and configure PHP on a server. It also discusses best practices like separating environments, using variables, reusing roles from Ansible Galaxy, and secrets management.
Why you should add React to your Rails application now!
React is a javascript library for building user interfaces which has many great features going for it. It also happens to be easy to introduce in small increments, particularly when compared to other javascript frameworks, and plays well with Rails. Find out how you can start modernizing the javascript in your Rails application today, without major refactoring.
Expressjs from-zero-to-hero
This document summarizes Express, a web application framework for Node.js. It notes that Express is minimalist, unopinionated, and fast, processing around 21k requests per second. It discusses key Express features like middleware for request/response processing, error handling, routing, and template engines. It provides examples of middleware, security, and input validation packages that can be used with Express. The overall document introduces Express and its core capabilities for building web applications and APIs.
It Sounded Good on Paper - Lessons Learned with Puppet
This talk is a 12 point guide on the things we did wrong during our journey with Puppet. We hope sharing helps people prevent the same mistakes in the future.
PuppetConf 2016: Puppet & Azure – Kenaz Kwa, Puppet
Here are the slides from Kenaz Kwa's PuppetConf 2016 presentation called Puppet & Azure. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
Sample From Ramesh
This document discusses the basics of using Git and GitHub for source control. It explains some key concepts like branches, committing changes locally, checking for updates from others, and pushing commits to a remote repository to share changes. The document walks through common Git commands like cloning a repository, checking out branches, committing, merging, and pushing to share code.
React, London JS Meetup, 11 Aug 2015
React is a revolution in web UI technology that uses a virtual DOM for fast rendering. It allows developers to build user interfaces as components in a declarative way using JavaScript or JSX. Components are fast, stateless, and their UI is a projection of data. The virtual DOM diffing algorithm efficiently detects changes and updates the real DOM. React promotes building user interfaces with unidirectional data flow and separating concerns into components rather than MVC.
The Ruby workflow
The document discusses a Ruby workflow that focuses on continuous integration and deployment. It is presented by Clemens Helm, a Codeship employee, and proposes a workflow involving developing on feature branches, creating pull requests to merge code to the master branch, and then automatically deploying to production after the code is merged. The workflow aims to get ideas to users quickly through short development cycles, encourage contributions, and ensure code quality while making deployments effortless. However, it notes that the pull request cycle can be a potential downside.
Build Nodejs APIs using Serverless
Serverless lets you focus on coding and testing instead of provisioning infrastructure, configuring web servers, debugging your configuration, managing security settings, and all the drudgery normally associated with getting an app up and running. In this session with, you’ll discover how to migrate an API of an existing app to Azure Functions. You’ll learn how to use Visual Studio Code and the Azure Functions extension to speed up your work. After this session, you’ll join the ranks of serverless developers.
Chef vs. Puppet in the Cloud: How Telepictures and MoneySuperMarket Do It
Doug Babst from Telepictures discussed how they migrated their infrastructure from using RightScript to manage servers in AWS to using Chef. They now define their infrastructure as code using Chef recipes which are version controlled and tested locally using Vagrant before being deployed to AWS. This allows them to more easily and consistently configure their 150 servers.
Jim Davies from MoneySupermarket discussed how they use Puppet to manage their infrastructure in a dynamic cloud environment. They take a masterless approach where instances pull their own configuration from a Git repository. They leverage tools like Facter, Hiera and Librarian-Puppet to parameterize configurations and deploy roles to instances. This infrastructure as code approach allows them to easily deploy their business services across different
IoT Google Cloud Functions with Firebase
Presentation slides used when introducting Google Cloud Functions, as well as Cloud Functions for Firebase, with Internet of Things
What's hot (20)
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
Maintenance Mode: Leveraging Chef to Schedule Patching, Reboot, etc.
PuppetConf 2017: No Server Left Behind - Miguel Di Ciurcio Filho, Instruct
PuppetConf 2017: No Server Left Behind - Miguel Di Ciurcio Filho, Instruct
PuppetConf 2017: Using Puppet Enterprise APIs with PowerShell- Jason Johnson,...
PuppetConf 2017: Using Puppet Enterprise APIs with PowerShell- Jason Johnson,...
Configuring SQL Server Reporting Services for ASP.NET Running on Azure Web Role
Configuring SQL Server Reporting Services for ASP.NET Running on Azure Web Role
From No Git to 3000 GitHub Users and How to Keep Them Happy - GitHub Universe...
From No Git to 3000 GitHub Users and How to Keep Them Happy - GitHub Universe...
Modern infrastructure as code with ansible cake fest 2021
Modern infrastructure as code with ansible cake fest 2021
Why you should add React to your Rails application now!
Why you should add React to your Rails application now!
It Sounded Good on Paper - Lessons Learned with Puppet
It Sounded Good on Paper - Lessons Learned with Puppet
PuppetConf 2016: Puppet & Azure – Kenaz Kwa, Puppet
PuppetConf 2016: Puppet & Azure – Kenaz Kwa, Puppet
Chef vs. Puppet in the Cloud: How Telepictures and MoneySuperMarket Do It
Chef vs. Puppet in the Cloud: How Telepictures and MoneySuperMarket Do It
Viewers also liked
Removing Barriers to Going Fast
The document discusses strategies for removing barriers to speed and continuous deployment at Etsy. It describes establishing a culture of open communication, trust, and removing fear of failure in order to optimize for minimizing time to recovery from incidents rather than minimizing incidents. Key techniques discussed include enabling anyone to deploy code, conducting blameless post-mortems, and using automation tools like Deployinator to streamline deployments.
Mean Time to Sleep: Quantifying the On-Call Experience
Starting an on-call rotation can be like opening a door into the unknown. You don’t know if it will be a bad week or if it will be an especially bad week. You don’t know what to expect. Thinking that historical information from past on-call rotations might yield useful insights, Etsy’s Operations team set out to quantify the on-call experience, identify what made it difficult, and use those data to reduce the incidence of pain points in an attempt to make being on call more bearable.
Open Source Monitoring Tools
This document summarizes and compares several open source monitoring tools: Nagios, Graphite, StatsD, Logstash, and Sensu. Nagios is introduced as a commonly used tool that some love and some find frustrating. Graphite is described as a tool for storing and graphing time-series data. StatsD aggregates counters and timers and sends them to backend services like Graphite. Logstash is an tool for managing logs and events that can input, filter, and output data. Sensu is a monitoring router that connects check scripts to handler scripts to alert or process monitoring data. Examples are given for each tool and what types of metrics to collect.
Ipv6 basics
There's no place like ::1 -- IPv6 Basics
Given at Etsy during a Lunch'n'Learn session on 2012-06-01.
A Whirlwind Tour of Etsy's Monitoring Stack
Etsy uses a variety of open source and custom tools to monitor their infrastructure including Ganglia, StatsD, Graphite, Syslog-Ng, Logster, Splunk, Logstash, Eventinator, Chef, Nagios, and Nagios Herald. They collect system, application, and log metrics and events and store them for analysis. Their monitoring setup is fully configured through Chef to provide consistent deployments across environments with over 120 monitoring recipes.
ELK: Moose-ively scaling your log system
This document discusses Etsy's journey with Elasticsearch, Logstash, and Kibana (ELK) over 3 years. It covers lessons learned in monitoring and scaling Logstash and Elasticsearch clusters. Key topics include sizing Elasticsearch and Logstash clusters based on resources like CPU, memory, disk I/O, and networking. Monitoring systems and metrics is also discussed. The document provides advice on optimizing Logstash performance by measuring baselines, managing garbage collection, and writing custom plugins when needed. Testing configuration changes is emphasized.
Creating a Data-Driven Organization: an executive summary
What does it mean for an organization to be data-driven? It is not about having lots of reports and dashboards or big data but having the right data culture. Learn more about that culture in this executive summary of the key findings in Carl Anderson's new book "Creating a Data-Driven Organization" (2015) from O'Reilly Media.
[devops REX 2016] Retour d’expérience de la transformation DevOps de Microsoft
Samuel Métias, Microsoft @ devops REX 2016
Samuel partage avec les participants de devops REX la transformation du fonctionnement des équipes Microsoft, et notamment des équipes produits (Office, Windows, Visual Studio) qui ont adopté une démarche DevOps. Cette (r)évolution, un vrai challenge pour une entreprise de la taille de Microsoft, constitue une démarche clé dans le processus de transformation digitale des organisations.
The DevOps Wonder @ PHPTour Lyon 2014
http://joind.in/talk/view/11245
Dans notre économie numérique, ce n’est pas les gros qui mangent les petits, c’est les rapides qui mangent les lents. Les méthodes de gestion de projets informatiques traditionnelles ont mené à des échecs spectaculaires en termes de délais et de gestion de risque. En parallèle, des leaders du web tels qu’Amazon, Netflix ou Google ont atteint une vélocité incroyable grâce à une implémentation audacieuse des principes d’agilité. Parmi ces différents mouvements agiles, DevOps rassemble des experts du développement et de l’opérationnel sur la manière dont doit être implémentée l’agilité, de la conception technique jusqu’à la mise en production, pour atteindre une vélocité maximale. Dans cette conférence, je partagerai l’expérience de Theodo sur plusieurs projets de grande envergure (jusqu’à 15 développeurs). Je présenterai ce que DevOps signifie pour nous et comment il nous aide à livrer nos projets de manière plus rapide et plus fiable. Nous passerons en revue les challenges auxquels nous avons été confrontée, tant d’un point de vue management, technique ou culturel et présenterons les solutions que nous avons trouvées, basées sur des technologies puissantes : Symfony2, OpenStack, Puppet, Vagrant, Capifony, Jenkins, Behat et d’autres…
SEO pour ETSY
En ouvrant une boutique en ligne sur ETSY on espère tous se faire voir par des millions de visiteurs et avoir des ventes des quatres coins du monde ! Mais il ne suffit pas seulement de placer quelques photos, inscrire deux ou trois mots clés et attendre que les ventes se fassent toutes seules ! Il faut travailler fort pour ressortir parmi les millions de fiches mises en ligne sur la plate-forme!
Et c'est là que les notions de SEO, d'Anaytics et de rédaction Web prennent leur sens.
En anglais c'est Search Engine Optimization, en français, on appelle ça du référencement naturel. C'est l'ensemble des techniques utilisées pour s'assurer de bien ressortir dans les pages des résultats de recherche.
Cette présentation vise les artistes et les artisans sur Etsy pour démystifier et simplifier de nombreux aspects qui sont pris en compte dans le positionnement des produits sur Etsy.
Viewers also liked (10)
Mean Time to Sleep: Quantifying the On-Call Experience
Mean Time to Sleep: Quantifying the On-Call Experience
Creating a Data-Driven Organization: an executive summary
Creating a Data-Driven Organization: an executive summary
[devops REX 2016] Retour d’expérience de la transformation DevOps de Microsoft
[devops REX 2016] Retour d’expérience de la transformation DevOps de Microsoft
Similar to Data Driven Monitoring
Using Riak for Events storage and analysis at Booking.com
At Booking.com, we have a constant flow of events coming from various applications and internal subsystems. This critical data needs to be stored for real-time, medium and long term analysis. Events are schema-less, making it difficult to use standard analysis tools.This presentation will explain how we built a storage and analysis solution based on Riak. The talk will cover: data aggregation and serialization, Riak configuration, solutions for lowering the network usage, and finally, how Riak's advanced features are used to perform real-time data crunching on the cluster nodes.
Cloud dwh
The majority of cloud-based DWH provides a wide range of migration tools from in-house DWH. However, I believe that cloud migration success is based not only on reducing infrastructure maintenance costs, but also on additional performance profit inherited from tailored data model.
I am going to prove that copying star or snowflake schemas as is will not lead to maximum performance boost in such DWH as Amazon Redshift and Google BigQuery. Moreover, this approach may cause additional cloud expenses.
We will discuss why data models should be different for each particular database, and how to get maximum performance from database peculiarities.
Most of performance tuning techniques for cloud-based DWH are about adding extra nodes to cluster, but it may lead to performance degradation in some cases, as well as extra costs burden. Sometimes, this approach allows to get maximum speed from current hardware configuration, may be even less expensive servers.
I will show some examples from production projects with extra performance using lower hardware, and edge cases like huge wide fact table with fully denormalized dimensions instead of classical star schema.
Building real-time data analytics on Google Cloud
Presented at the 25th Data Science Leuven meetup on 2020/03/11
Jonny Daenen explains the steps they took at Selligent to create a multi-tenant real-time data pipeline. He discusses all challenges the team encountered as well as the tools they used. The benefits of using Google Cloud Platform to remove operational hurdles when moving data pipelines to production are demonstrated.
Productionizing a Machine Learning System at a Large Australian Telco with Ca...
Many organisations face the difficult challenge of enabling Machine Learning projects to get to market more quickly and to allow data science teams to share their feature. In this talk, I will be discussing the machine learning pipeline developed at one of Australia’s largest telecommunications companies to achieve this goal using Spark and Spark ML as well as the challenges faced along the way. I’ll begin by discussing the utility and motivation for a centralised feature store, before looking at the complexities of such an undertaking (both technical and otherwise).
We’ll then dig into the technical details of implementation by discussing the scalability headaches we faced and dive into the details of the solutions used to drastically improve the speed and organisational scalability of the system. Several areas that will be covered are providing a declarative API that allowed us to compile feature definitions into optimised spark code, tuning the spark DAG for drastically improved parallelism, adjusting the workflow for different machine learning use cases and fine tuning the resource allocation to avoid unnecessary bottlenecks.
Finally we will touch on lessons learnt along the way and offer advice on things to avoid as well as how to take things to the next level. Attendees will leave with a better understanding of the challenges faced with scaling a centralised machine learning pipeline at a large organisation. They will also have plenty of real world practical optimisation techniques that they can apply to their own problems.
Big Data Science Challenges in Media
Big data science faces challenges related to volume, variety, and velocity of data. Specifically:
- Petabytes of structured and unstructured data from 10 million subscribers across 10 touchpoints is too big for traditional data warehouses. This includes event logs, program data, content metadata, and purchase histories.
- The data comes in at 140 MB/s (12 TB/day), which is too fast as 95% of the data is dropped currently.
- The data has inconsistent structures, fragmented XML, and shifting fields, making it difficult to extract features and implement machine learning algorithms at scale.
Cloud DWH deep dive
The presentation cover architecture on main cloud dwh players as google big query, aws redshift and snowflake as well as data modeling pecularities
Hadoop Migration from 0.20.2 to 2.0
Hadoop was upgraded from version 0.20.2 to 2.0 across 5 production clusters storing over 1 petabyte of data and processing billions of events daily. Key challenges included managing configurations across heterogeneous clusters, allowing data movement between clusters on different versions, ensuring application compatibility, addressing capacity limitations, and performing the upgrade with minimal downtime. Extensive testing, parallel component upgrades, and rolling restarts were used to successfully complete the upgrade while maintaining over 99% uptime and without any failures.
a real-time architecture using Hadoop and Storm at Devoxx
The document discusses a real-time architecture using Hadoop and Storm. It proposes a layered architecture with a batch layer using Hadoop for large-scale immutable data processing, a speed layer using Storm for continuous processing of incoming data, and a serving layer to merge results from the batch and real-time layers for queries. The architecture is based on an event-driven, immutable data model and aims to provide low-latency queries over all data through real-time and batch views.
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (Presented as part of the "Batte of the Data Transformation Tools" Learning Path at PASS Data Community Summit on November 15th, 2023)
Life in the Fast Lane: Speed, Usability & Search Engine Optimization
In 2010, Google announced that site speed was a ranking factor and the reaction was a collective shrug. When you load your site in Chrome, it’s quick, right? So why worry? Mobile. We know that you bought the iPhone 5 as soon as it came out, but what about the poor sucker who’s still on a Blackberry? Did you know that every added second on load time on a mobile device causes 20% of your audience to bail? You can be #1 on Google, but if your site takes 10 seconds to load, you might as well be #10,000. And responsive design won’t save you. In this seminar, learn tactics to make your WordPress site blazing fast, including responsive designs. It’s speed, usability and SEO, mashed together in delicious harmony.
Integrating multiple CDN providers at Etsy - Velocity Europe (London) 2013
The document discusses Etsy's experience integrating multiple content delivery network (CDN) providers. Etsy began using a single CDN in 2008 but then investigated using multiple CDNs in 2012 to improve resilience, flexibility, and costs. They developed an evaluation criteria and testing process to initially configure and test the CDNs with non-critical traffic before routing production traffic. Etsy then implemented methods for balancing traffic across CDNs using DNS and monitoring the performance of the CDNs and origin infrastructure.
stackconf 2022: Infrastructure Automation (anti) patterns
Over the past 1.5 decade our industry has tried to adopt an increased amount of infrastructure automation. We called it Configuration Management, Infrastructure as Code, infrastructure as Software, Provisioning, Orchestration. We learned about Desired State, Idempotence, etc.. We have seen a number of tools become popular; we have seen a number of tools disappear. But over the years we have seen a number of patterns appear and reappear. Patterns that lead to actually getting great benefits out of automation, or just wasting time while missing out on goals. This talk will explain you a number of these patterns which we have frequently encountered in the wild, with their benefits and caveats. We will try to keep this tool agnostic. Your vision might be Clouded, and you might have to take this with a grain of Salt while you play the Chef from the Muppet show the story, all names, characters, and incidents portrayed in this production are fictitious. No identification with actual persons (living or deceased), places, buildings, and products are intended or should be inferred.
Infrastructure as Code Patterns
StackConf 2022 Talk about The Different Patterns and AntiPatterns
we've seen over the past decade of Automation
Top .NET, Java & Web Performance Mistakes - Meetup Jan 2015
Top .NET, Java & Web Performance Problems. Why these apps failed, how to avoid it and which metrics to look at, e.g: # of Busy vs. Idle Worker Threads, Connection Pool Acquisition Time, # Exceptions Thrown, ...
Rubyslava + PyVo #48
Skypicker is a flight ticket search and booking engine that processes hundreds of terabytes of airline data monthly and sells thousands of tickets daily in Europe, Russia, and China. It uses PostgreSQL extensively to power its API and databases. The document discusses Skypicker's PostgreSQL infrastructure, which includes 5 database clusters with over 0.5TB of memory each that handle 20 million updates per hour. It also covers optimizations made to scale the databases, such as table partitioning, cascading replication, and query tuning. Skypicker is currently preparing for a large increase in data and is hiring database masters and developers.
dbt Python models - GoDataFest by Guillermo Sanchez
Guillermo Sanchez presented on the pros and cons of using Python models in dbt. While Python models allow for more advanced analytics and leveraging the Python ecosystem, they also introduce more complexity in setup and divergent APIs across platforms. Additionally, dbt may not be well-suited for certain use cases like ingesting external data or building full MLOps pipelines. In general, Python models are best for the right analytical use cases, but caution is needed, especially for production environments.
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
Pairing your real-time operational data stored in a modern database like MongoDB with first-class business intelligence platforms like Tableau enables new insights to be discovered faster than ever before.
Many leading organizations already use MongoDB in conjunction with Tableau including a top American investment bank and the world’s largest airline. With the Connector for BI 2.0, it’s never been easier to streamline the connection process between these two systems.
In this webinar, we will create a live connection from Tableau Desktop to a MongoDB cluster using the Connector for BI. Once we have Tableau Desktop and MongoDB connected, we will demonstrate the visual power of Tableau to explore the agile data storage of MongoDB.
You’ll walk away knowing:
- How to configure MongoDB with Tableau using the updated connector
- Best practices for working with documents in a BI environment
- How leading companies are using big data visualization strategies to transform their businesses
systemd @ Facebook -- a year later
This talk is a followup to Deploying systemd at scale that was presented at systemd.conf 2016, and covers the aftermath of the migration of our fleet to CentOS 7. Now that systemd is available everywhere, we found more and more services that started adopting it for their deployment, leveraging its features and occasionally exposing interesting behaviors. At the same time, we've been able to hone our process for integrating and rolling out new versions of systemd on the fleet, and started building tooling to manage and monitor it at scale.
DDD in Pixel Federation volume 2
- The document discusses how a game company called Pixelfederation handles data in their games using domain-driven design principles. It covers how they process live game data and balancing data across multiple databases, using database abstraction and doctrine to maintain consistency. While DDD best practices suggest transactions within a single aggregate, the company found it better to expand transaction boundaries for multiple aggregates due to their technical constraints and for user experience.
Getting 100B Metrics to Disk
Awesome! Traffic to your site is really picking up and everything is lookin’ good. Well, except for that database back in the corner, but it will hold… right? No one really wants to deal with scaling the database tier, but hopefully your customers will drag you (perhaps kicking and screaming) to some sort of distributed database architecture.
This talk is all about scaling MySQL through hardware optimizations and sharding from a Site Engineering perspective. This includes real world examples of finding pain points, identifying risks, and evaluating cloud vs hardware scaling. I’ll also discus distributed database management, dealing with data purging, making consistent backups, and how to keep the site up when things go bad.
Similar to Data Driven Monitoring (20)
Using Riak for Events storage and analysis at Booking.com
Using Riak for Events storage and analysis at Booking.com
Productionizing a Machine Learning System at a Large Australian Telco with Ca...
Productionizing a Machine Learning System at a Large Australian Telco with Ca...
a real-time architecture using Hadoop and Storm at Devoxx
a real-time architecture using Hadoop and Storm at Devoxx
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...
Life in the Fast Lane: Speed, Usability & Search Engine Optimization
Life in the Fast Lane: Speed, Usability & Search Engine Optimization
Integrating multiple CDN providers at Etsy - Velocity Europe (London) 2013
Integrating multiple CDN providers at Etsy - Velocity Europe (London) 2013
stackconf 2022: Infrastructure Automation (anti) patterns
stackconf 2022: Infrastructure Automation (anti) patterns
Top .NET, Java & Web Performance Mistakes - Meetup Jan 2015
Top .NET, Java & Web Performance Mistakes - Meetup Jan 2015
dbt Python models - GoDataFest by Guillermo Sanchez
dbt Python models - GoDataFest by Guillermo Sanchez
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
More from Daniel Schauenberg
Human Factors and PostMortems
Our daily work takes place in a myriad of systems. They are comprised of software, hardware and humans. And everybody who has worked with complex systems at any scale knows: Failure is not an option, it's inevitable. At Etsy we are embracing the fact that failures happen and that the only way to understand how the accident happened is to investigate it without blaming the humans involved. This is why we have a blameless postmortem for every outage that occurs. It is an open meeting and everybody is invited to join and find out what happened and how we can make the system safer. This talk will explain how postmortems at Etsy are conducted and how we maintain and scale the process as the team grows and new people start. It will go over the tools we built and utilize to make postmortems efficient and also share the learnings from each one with all the people in the company.
Deploy, Collaborate and Listen
At Etsy we are continuously deploying software with about 50 deploys per day on average. We have about 150 engineers that work together to add features, improve the site, solve problems, and figure out outages. We are constantly working on improving ways to collaborate and have the time to invest into research and new projects instead of fire fighting. However this hasn't been like this from the beginning. In the Dark Ages of Etsy we had a vastly different software architecture, a myriad of silos with a less than ideal amount of communication and a not so stable site. So what happened? How did we fix this? We will briefly go over how we changed the architecture and culture to make site operations and stability better. After that we will go into detail about what it meant for us to maintain that culture of collaboration and trust and things we learned from that. And finally take a look at the state of things in the present and give a comprehensive picture of how we arrived here and where to go from there.
Development, Deployment and Collaboration at Etsy
At Etsy about 150 engineers deploy a single monolithic application more than 60 times a day. This process of deploying small changesets continuously enables us to build up and release robust features and detect and fix bugs extremely fast. All while serving over a billion page views per month. Developing and deploying at such a high velocity however only works because product developers and designers, infrastructure and operations engineers and the security team work closely together. We have an extremely open culture of sharing (inside and outside the company) and make sure we run into as few surprises as possible by bringing everybody on the same page about changes.
Scaling Deployment at Etsy
How deployment is tackled at Etsy and how we keep it fast and scale it out with a growing engineering team.
Reprised version of the talk given at the Continuous Delivery NYC meetup October 2013.
Feature Flagging your Infrastructure for Fun and Profit
At Etsy we run our production stack exclusively on physical hardware and have (including developer and CI VMs) about 1000 nodes managed with a single Chef server running the same cookbooks across development and production nodes. We have built the knife-spork plugin to support our workflow and have 30 engineers making changes about 20 times a day. While this setup keeps configuration changes and package installations consistent across our network, it makes it harder to gradually roll out and test changes on a per node or role basis. In the talk our general workflow will be introduced and explained and how we enable engineers to frequently roll out small changes. Following the general introduction, our workflow for testing changes will be discussed and the library we wrote to enable config flags and gradual roll outs for our infrastructure within our chef recipes.
DevTools at Etsy
The DevTools team at Etsy consists of 5 engineers who build and maintain tools to help the 150 committers at Etsy. These tools include Deployinator for deploying code, DeveloperVMs for local development environments, Jenkins slaves for continuous integration, DevDB Proxy for accessing production data safely, and various monitoring and debugging tools like StatsD, Logster, Graphite, and Supergrep. The DevTools team works to foster collaboration between development and operations through these shared tools.
Scaling Deployment at Etsy
Etsy has scaled its deployment process to support around 150 engineers. Key aspects include running continuous integration tests across many virtual machines to avoid bottlenecks, coordinating deployments through an IRC channel with scheduled "push trains", and tools like Deployinator that automate deployments and ensure code quality and version consistency. The goal is to constantly improve deployment speed while maintaining robustness, bring development environments closer to production, and treat inability to deploy with the same urgency as a site outage.
StatsD Workshop Monitorama 2013
This document summarizes a StatsD workshop presented by Daniel Schauenberg at Monitorama 2013. The workshop covered what StatsD is, the different types of metrics it can handle, and how to set up and configure StatsD and its backends like Graphite. It also provided examples of sending metrics to StatsD and ideas for extending and customizing StatsD functionality.
Etsy chef-workflow
Etsy uses Chef for infrastructure configuration and management. They have around 1000 nodes managed by around 30 devops engineers making regular changes. Chef Server and GitHub Enterprise are used to manage configuration code and changes. Knife-spork is used as the deployment tool to promote changes through development, staging, and production environments. Monitoring and notification tools like IRC alerts are used for change management and failure reporting.
More from Daniel Schauenberg (9)
Feature Flagging your Infrastructure for Fun and Profit
Feature Flagging your Infrastructure for Fun and Profit
Recently uploaded
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/how-axelera-ai-uses-digital-compute-in-memory-to-deliver-fast-and-energy-efficient-computer-vision-a-presentation-from-axelera-ai/
Bram Verhoef, Head of Machine Learning at Axelera AI, presents the “How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-efficient Computer Vision” tutorial at the May 2024 Embedded Vision Summit.
As artificial intelligence inference transitions from cloud environments to edge locations, computer vision applications achieve heightened responsiveness, reliability and privacy. This migration, however, introduces the challenge of operating within the stringent confines of resource constraints typical at the edge, including small form factors, low energy budgets and diminished memory and computational capacities. Axelera AI addresses these challenges through an innovative approach of performing digital computations within memory itself. This technique facilitates the realization of high-performance, energy-efficient and cost-effective computer vision capabilities at the thin and thick edge, extending the frontier of what is achievable with current technologies.
In this presentation, Verhoef unveils his company’s pioneering chip technology and demonstrates its capacity to deliver exceptional frames-per-second performance across a range of standard computer vision networks typical of applications in security, surveillance and the industrial sector. This shows that advanced computer vision can be accessible and efficient, even at the very edge of our technological ecosystem.AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Principle of conventional tomography-Bibash Shahi ppt..pptx
before the computed tomography, it had been widely used.
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Dmitrii Kamaev, PhD
Senior Product Owner - QIAGEN
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots - like ChatGPT - can converse on any topic, task-oriented chatbots - the focus of this paper - are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots.
To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Recently uploaded (20)
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
“How Axelera AI Uses Digital Compute-in-memory to Deliver Fast and Energy-eff...
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Data Driven Monitoring
- 1. Data Driven Monitoring Daniel Schauenberg d@etsy.com @mrtazz
- 3. @mrtazz
- 6. @mrtazz
- 7. How comfortable are you deploying a change right now?
- 8. “If this is your first day at Etsy, you deploy the site”
- 11. @mrtazz
- 12. @mrtazz Ganglia • System level metrics • Instance per DC/environment • > 220k RRD files • Fully configured through Chef role attributes
- 14. @mrtazz StatsD
- 15. @mrtazz Graphite • Application level metrics • 96G RAM, 20 Cores, 7.3T SSD RAID 10 • 525k metrics/minute • Mirrored Primary/Primary Setup • Functionally sharded relays
- 16. @mrtazz
- 17. @mrtazz
- 18. @mrtazz nagios
- 20. @mrtazz
- 21. @mrtazz Nagios • 2 instances in each DC/environment • Fully Chef generated configuration • Service checks and contacts in git • Notifications via email->SMS gateway • ~75% ops on-call
- 23. @mrtazz
- 24. @mrtazz Much more… • Syslog-ng • Logstash • Logster • Supergrep • Eventinator
- 25. Information Overload Image by http://jasoncasteel.deviantart.com/
- 27. We have the data We can make it better Item by PicksFromThePast
- 32. @mrtazz Failed Check nagios-herald Formatter Helpers Message Graphite Ganglia Logstash
- 36. @mrtazz
- 39. @mrtazz Wearables! Item by JennysTrinketShoppe
- 42. @mrtazz Summary • Set of trusted tools for monitoring • Always experiment • Always learn • Always improve • Use the data, Luke
- 43. @mrtazz Shout out to @lozzd and @Ryan_Frantz
- 44. @mrtazz codeascraft.com etsy.com/codeascraft/talks etsy.github.com etsy.com/careers
- 45. Questions?
- 46. Data Driven Monitoring Daniel Schauenberg d@etsy.com @mrtazz