Latency: The Silent Monitoring System Killer

•

2 likes•1,761 views

Lindsay Holmwood

Slides from a lighting talk presented at the January Sydney DevOps Meetup.

LATENCY
THE SILENT
MONITORING SYSTEM
KILLER

Saturday, 21 January 2012

#MONITORINGSUCKS

Saturday, 21 January 2012

execute large volumes of monitoring
checks under a variety of conditions
(good + bad) with a consistent throughput

Saturday, 21 January 2012

CONSISTENT
THROUGHPUT

Saturday, 21 January 2012

SHORT CHECK
EXECUTION TIME

Saturday, 21 January 2012

WHAT
INTRODUCES
VARIABILITY?

Saturday, 21 January 2012

LATENCY INTRODUCED
c SYNCHRONOUS CALLS

Saturday, 21 January 2012

= 1 monitoring check

Saturday, 21 January 2012

150 monitoring checks
each executed every 300 seconds
each takes 1 second
checks are executed serially

Saturday, 21 January 2012

all checks executed in 150 seconds
monitoring system at 50% capacity

Saturday, 21 January 2012

DOUBLE THE CHECKS

Saturday, 21 January 2012

all checks executed in 300 seconds
monitoring system at 100% capacity

Saturday, 21 January 2012

DOUBLE THE
EXECUTION TIME

Saturday, 21 January 2012

all checks executed in 600 seconds
monitoring system at 200% capacity
only 50% of the checks are “on time”

Saturday, 21 January 2012

CHECK LATENCY

Saturday, 21 January 2012

HOW DO
WE FIX THIS!?

Saturday, 21 January 2012

PERFORMANCE
ANALYSIS!

Saturday, 21 January 2012

MONITORING CHECK
==
“ACTION” ON MVC WEB APP

Saturday, 21 January 2012

SEPARATE DATA
COLLECTION
FROM

THRESHOLDING
f NOTIFICATIONS

Saturday, 21 January 2012

THIS SHIFTS
LATENCY

Saturday, 21 January 2012

IT DOES NOT
ELIMINATE IT!

Saturday, 21 January 2012

RRDTOOL IS EVIL

Saturday, 21 January 2012

USE SOMETHING
BETTER!

Saturday, 21 January 2012

USE
OPENTSDB
OR

GANGLIA
WITH
CHECK_TSDB
OR

CHECK_GMOND
Saturday, 21 January 2012

A DIFFERENT SET a
PROBLEMS

Saturday, 21 January 2012

STORAGE WILL
GO AWAY

Saturday, 21 January 2012

CHAOS WILL ENSUE

Saturday, 21 January 2012

PAGERS WILL MELT

Saturday, 21 January 2012

SET UP “META-
PARENTING”

Saturday, 21 January 2012

BUILD A KILL
SWITCH

Saturday, 21 January 2012

READ MORE
ABOUT THIS:

Saturday, 21 January 2012

bit.ly/yN4mdy

Saturday, 21 January 2012

On January 31, 2000, Alaska Airlines Flight 261 plunged into the Pacific ocean in an extreme "nose down" position, killing all 88 crew and passengers on board. The NTSB concluded AA261's horizontal stabiliser trim system's jackscrew was inadequately maintained, causing the pilots to lose all control of the plane. There are striking parallels with the problems we face daily in IT operations & software development, and the 30 years of give and take between the aircraft manufacturer's engineers, airline maintenance staff, and federal regulators that preceded AA261's simple mechanical failure. In this talk, Lindsay looks at the complex interplay between the parties in the AA261 crash through a DevOps lens, investigating the collaborative approach to maintenance and operation of the MD-83 aircraft, and relating the complexities back to the complex IT systems we build and maintain.

Islands: Puppet at Bulletproof Networks

Lindsay Holmwood

Bulletproof Networks provides managed hosting services to some of the largest companies in Australia. Bulletproof implements strong isolation of customer environments, and this can present unique challenges when re-using Puppet code across our customer base. Additionally, the environments range in size from small to very large, and our tools + processes need to be able to handle both uses cases equally well. In this talk Lindsay + Mick will cover how Bulletproof's approach to these problems has evolved over the last 4 years, and some of the tools Bulletproof has developed and built upon to provide an awesome service to our customers.

Escalating complexity: DevOps learnings from Air France 447

Lindsay Holmwood

On June 1, 2009 Air France 447 crashed into the Atlantic ocean killing all 228 passengers and crew. The 15 minutes leading up to the impact were a terrifying demonstration of the how thick the fog of war is in complex systems. Mainstream reports of the incident put the blame on the pilots - a common motif in incident reports that conveniently ignore a simple fact: people were just actors within a complex system, doing their best based on the information at hand. While the systems you build and operate likely don't control the fate of people's lives, they share many of the same complexity characteristics. Dev and Ops can learn an abundance from how the feedback loops between these aviation systems are designed and how these systems are operated. In this talk Lindsay will cover what happened on the flight, why the mainstream explanation doesn't add up, how design assumptions can impact people's ability to respond to rapidly developing situations, and how to improve your operational effectiveness when dealing with rapidly developing failure scenarios.

Load testing with Blitz

Lindsay Holmwood

We all know that load testing is important, but it's all too common that it's left to the very end of a project and it's invariably the first thing that gets dropped when budgets and timeframes get cut. Furthermore, most of us don't know where or how to start implementing effective load tests, let alone how to analyse the results. Lindsay Holmwood, Software Manager at Bulletproof Networks, will be talking about integrating performance testing into your application development + deploy cycle from the very beginning, using inexpensive and easy to use SaaS tools. There will be a hands on demonstration of the Blitz load + performance testing tool, coupled with a brief dive into the Blitz API internals to retrieve and analyse advanced reporting information.

Behaviour Driven Monitoring with cucumber-nagios

Lindsay Holmwood

Writing checks for your monitoring system is boring. You end up writing the same checks again and again, and it can be difficult to verify behavior instead of availability. Wouldn't it be useful to have a standard library of checks you could reuse across your infrastructure? it lets you write reusable behavioral tests in human-readable language.Say hello to cucumber-nagios - it lets you write reusable behavioral tests in human-readable language. As cucumber-nagios output the test results in the Nagios plugin format you can run your checks from any monitoring system that understands the format.

Rump - making Puppetmaster-less Puppet meaty

Lindsay Holmwood

We’re all familiar with the Puppet manifest development cycle: “make your change, commit your change, push your change, run Puppet, debug the output, then go back to the beginning to refactor or fix a bug”. What if you could could shorten the cycle to “make a change, run puppet, commit change”? Enter Rump, a tool for doing Puppet runs locally from a Git checkout. Rump encourages a Puppet workflow where you quickly + iteratively develop your Puppet manifests on a single machine, then push your changes up to a repository to deploy to the rest of your infrastructure. Lindsay will be demonstrating how to setup Rump, showing off some awesome workflows for doing local iterative development, and exposing super cool features like testing your manifests against new versions of Puppet in a single command.

Behaviour driven infrastructure

Lindsay Holmwood

Does Behaviour Driven Development have a role in the infrastructure world? Enter Behaviour Driven Infrastructure where systems administrators can apply some simple open source tools and BDD principles to make infrastructure management more powerful, more insightful and deliver more value to their customers. The typical enterprise monitoring and configuration management set-up for a website is: - Is the Apache package installed and the appropriate version? - Is the Apache service running? - Can I connect to the HTTP port and is HTML returned? - Multiply this by a few hundred iterations of hosts and types of services and you’re probably looking at your typical Nagios, Puppet, Cfengine, Hyperic set-up. All this monitoring misses something critical though – we’re not actually monitoring that the service does what it should. Yes, it matters whether Apache is installed, the Apache service is running, and you can connect to HTTP but does this actually prove anything about the availability of the service we’re managing and providing for our customers? Nope… We need to demonstrate more than just a check that says the Apache server is up. We need to demonstrate that the service delivered by that infrastructure was available to our customers AND functioning as intended. Enter Behaviour Driven Infrastructure or BDI which applies the principles of Behavioural Driven Development to the management of infrastructure. In this presentation you’ll learn: - How Behaviour Driven Development works - What makes a behavioural test - How to install and use Cucumber to perform BDI - Practical examples of behavioural tests in Cucumber, and - How to integrate BDI into your work flow and your enterprise monitoring and configuration management frameworks.

Burn down the silos! Helping dev and ops gel on high availability websites

Lindsay Holmwood

HA websites are where the rubber meets the road - at 200km/h. Traditional separation of dev and ops just doesn't cut it. Everything is related to everything. Code relies on performant and resilient infrastructure, but highly performant infrastructure will only get a poorly written application so far. Worse still, root cause analysis in HA sites will more often than not identify problems that don't clearly belong to either devs or ops. The two options are collaborate or die. This talk will introduce 3 core principles for improving collaboration between operations and development teams: consistency, repeatability, and visibility. These principles will be investigated with real world case studies and associated technologies audience members can start using now. In particular, there will be a focus on: - fast provisioning of test environments with configuration management - reliable and repeatable automated deployments - application and infrastructure visibility with statistics collection, logging, and visualisation

Flapjack: rethinking monitoring for the cloud

Lindsay Holmwood

Monitoring web application behaviour with cucumber-nagios

Lindsay Holmwood

Setting up monitoring for web applications can be complicated - tests tend to lack expressiveness, or and quite often they don't even test the right problem in the first place. cucumber-nagios lets a sysadmin write behavioural tests for their web apps in plain English, and outputs the test results in the Nagios plugin format, allowing a sysadmin to be notified by Nagios when their production apps aren't behaving.

Deploying Merb

Lindsay Holmwood

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

The Future of Platform Engineering

Jemma Hussein Allen

"Impact of front-end architecture on development cost", Viktor Turskyi

Fwdays

I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Search and Society: Reimagining Information Access for Radical Futures

Bhaskar Mitra

The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Bits & Pixels using AI for Good.........

Alison B. Lowndes

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

Viewers also liked

Your own (little) gem: building an online business with RubyLindsay Holmwood

AA261: DevOps lessons in collaborative maintenance

Lindsay Holmwood

Islands: Puppet at Bulletproof Networks

Lindsay Holmwood

Escalating complexity: DevOps learnings from Air France 447

Lindsay Holmwood

Load testing with Blitz

Lindsay Holmwood

Behaviour Driven Monitoring with cucumber-nagios

Lindsay Holmwood

Viewers also liked (6)

Your own (little) gem: building an online business with Ruby

AA261: DevOps lessons in collaborative maintenance

Islands: Puppet at Bulletproof Networks

Escalating complexity: DevOps learnings from Air France 447

Load testing with Blitz

Behaviour Driven Monitoring with cucumber-nagios

Recently uploaded

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

The Future of Platform Engineering

Jemma Hussein Allen

"Impact of front-end architecture on development cost", Viktor Turskyi

Fwdays

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Search and Society: Reimagining Information Access for Radical Futures

Bhaskar Mitra

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Bits & Pixels using AI for Good.........

Alison B. Lowndes

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

DevOps and Testing slides at DASA Connect

Kari Kakkonen

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Recently uploaded (20)

FIDO Alliance Osaka Seminar: Overview.pdf

The Future of Platform Engineering

"Impact of front-end architecture on development cost", Viktor Turskyi

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Neuro-symbolic is not enough, we need neuro-*semantic*

Search and Society: Reimagining Information Access for Radical Futures

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Bits & Pixels using AI for Good.........

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

UiPath Test Automation using UiPath Test Suite series, part 3

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

How world-class product teams are winning in the AI era by CEO and Founder, P...

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Connector Corner: Automate dynamic content and events by pushing a button

DevOps and Testing slides at DASA Connect

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

Latency: The Silent Monitoring System Killer

1. LATENCY THE SILENT MONITORING SYSTEM KILLER Saturday, 21 January 2012

2. #MONITORINGSUCKS Saturday, 21 January 2012

3. SCALABILITY Saturday, 21 January 2012

4. execute large volumes of monitoring checks under a variety of conditions (good + bad) with a consistent throughput Saturday, 21 January 2012

5. CONSISTENT THROUGHPUT Saturday, 21 January 2012

6. SHORT CHECK EXECUTION TIME Saturday, 21 January 2012

7. WHAT INTRODUCES VARIABILITY? Saturday, 21 January 2012

8. LATENCY INTRODUCED c SYNCHRONOUS CALLS Saturday, 21 January 2012

9. Saturday, 21 January 2012

10. = 1 monitoring check Saturday, 21 January 2012

11. Saturday, 21 January 2012

12. 150 monitoring checks each executed every 300 seconds each takes 1 second checks are executed serially Saturday, 21 January 2012

13. all checks executed in 150 seconds monitoring system at 50% capacity Saturday, 21 January 2012

14. DOUBLE THE CHECKS Saturday, 21 January 2012

15. Saturday, 21 January 2012

16. all checks executed in 300 seconds monitoring system at 100% capacity Saturday, 21 January 2012

17. DOUBLE THE EXECUTION TIME Saturday, 21 January 2012

18. Saturday, 21 January 2012

19. all checks executed in 600 seconds monitoring system at 200% capacity only 50% of the checks are “on time” Saturday, 21 January 2012

20. CHECK LATENCY Saturday, 21 January 2012

21. Saturday, 21 January 2012

22. HOW DO WE FIX THIS!? Saturday, 21 January 2012

23. PERFORMANCE ANALYSIS! Saturday, 21 January 2012

24. AN ANALOGY Saturday, 21 January 2012

25. MONITORING CHECK == “ACTION” ON MVC WEB APP Saturday, 21 January 2012

26. Saturday, 21 January 2012

27. Saturday, 21 January 2012

28. SEPARATE DATA COLLECTION FROM THRESHOLDING f NOTIFICATIONS Saturday, 21 January 2012

29. Saturday, 21 January 2012

30. THIS SHIFTS LATENCY Saturday, 21 January 2012

31. IT DOES NOT ELIMINATE IT! Saturday, 21 January 2012

32. Saturday, 21 January 2012

33. RRDTOOL IS EVIL Saturday, 21 January 2012

34. USE SOMETHING BETTER! Saturday, 21 January 2012

35. USE OPENTSDB OR GANGLIA WITH CHECK_TSDB OR CHECK_GMOND Saturday, 21 January 2012

36. Saturday, 21 January 2012

37. A DIFFERENT SET a PROBLEMS Saturday, 21 January 2012

38. STORAGE WILL GO AWAY Saturday, 21 January 2012

39. CHAOS WILL ENSUE Saturday, 21 January 2012

40. PAGERS WILL MELT Saturday, 21 January 2012

41. SET UP “META- PARENTING” Saturday, 21 January 2012

42. BUILD A KILL SWITCH Saturday, 21 January 2012

43. Saturday, 21 January 2012

44. READ MORE ABOUT THIS: Saturday, 21 January 2012

45. bit.ly/yN4mdy Saturday, 21 January 2012

Latency: The Silent Monitoring System Killer

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (6)

More from Lindsay Holmwood

More from Lindsay Holmwood (6)

Recently uploaded

Recently uploaded (20)

Latency: The Silent Monitoring System Killer