Mike Julian presented on building a more effective monitoring environment. He discussed three key aspects: making the monitoring system automated through configuration management and service discovery; reducing noise by implementing dependencies, handling redundant services, and using dynamic thresholds; and dynamically notifying stakeholders through tools like PagerDuty that allow flexible notification schedules. He provided examples of how to implement these concepts and recommended specific open source tools like collectd and Graphite that can help achieve effective monitoring.
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
Monitoring can mean very different things to different people, and this often leads to confusion and misunderstandings. There are many offerings both free software and commercials, and it's not always clear where each fits in the bigger picture. This talk will look a bit at the history of monitoring, and then into the general categories of Metrics, Logs, Profiling and Distributed tracing and how each of these is important in Cloud-based environment.
Video: https://www.youtube.com/watch?v=hCBGyLRJ1qo
Using examples, Rikard will discuss how serendipity is useful in testing and give ideas about how to introduce more good fortune in your organisation.
View on-demand webinar here - https://testhuddle.com/resource/good-testers-are-often-lucky-using-serendipity-in-software-testing/
How ransomware works and why it is breaching organisational defences
The best methods for prevention
The incident and crisis management & escalation process
A step-by-step guide to recovery
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
Monitoring can mean very different things to different people, and this often leads to confusion and misunderstandings. There are many offerings both free software and commercials, and it's not always clear where each fits in the bigger picture. This talk will look a bit at the history of monitoring, and then into the general categories of Metrics, Logs, Profiling and Distributed tracing and how each of these is important in Cloud-based environment.
Video: https://www.youtube.com/watch?v=hCBGyLRJ1qo
Using examples, Rikard will discuss how serendipity is useful in testing and give ideas about how to introduce more good fortune in your organisation.
View on-demand webinar here - https://testhuddle.com/resource/good-testers-are-often-lucky-using-serendipity-in-software-testing/
How ransomware works and why it is breaching organisational defences
The best methods for prevention
The incident and crisis management & escalation process
A step-by-step guide to recovery
An Introduction to Prometheus (GrafanaCon 2016)Brian Brazil
Often what you monitor and get alerted on is defined by your tools, rather than what makes the most sense to you and your organisation. Alerts on metrics such as CPU usage which are noisy and rarely spot real problems, while outages go undetected. Monitoring systems can also be challenging to maintain, and overall provide a poor return on investment.
In the past few years several new monitoring systems have appeared with more powerful semantics and which are easier to run, which offer a way to vastly improve how your organisation operates and prepare you for a Cloud Native environment. Prometheus is one such system. This talk will look at the monitoring ideal and how whitebox monitoring with a time series database, multi-dimensional labels and a powerful querying/alerting language can free you from midnight pages.
Evolution of Monitoring and Prometheus (Dublin 2018)Brian Brazil
This talk looks at the evolution of monitoring over time, the ways in which you can approach monitoring, where Prometheus fit into all this, and how Prometheus itself has grown over time.
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Brian Brazil
Often what you monitor and get alerted on is defined by your tools, rather than what makes the most sense to you and your organisation. Alerts on metrics such as CPU usage which are noisy and rarely spot real problems, while outages go undetected. Monitoring systems can also be challenging to maintain, and overall provide a poor return on investment.
In the past few years several new monitoring systems have appeared with more powerful semantics and which are easier to run, which offer a way to vastly improve how your organisation operates Prometheus is one such system. This talk will look at the monitoring ideal and how whitebox monitoring with a time series database, multi-dimensional labels and a powerful querying/alerting language can free you from midnight pages.
Prometheus is a next-generation monitoring system. It lets you see you not just what your systems look like from the outside, but also gives visibility into the internals and business aspects of your systems. This allows everyone to benefit, including both operations and developers. This talk will look at the concepts behind monitoring with Prometheus, how it's designed, why it's suitable for Cloud Native environments and how you can get involved.
Observability for Emerging Infra (what got you here won't get you there)Charity Majors
Distributed systems, microservices, containers and schedulers, polyglot persistence .. modern infrastructure patterns are fluid and dynamic, chaotic and transient. So why are we still using LAMP-stack era tools to debug and monitor them? We'll cover some of the many shortcomings of traditional metrics and logs (and APM tools backed by metrics or logs), and show how complexity is their kryptonite. So how do we handle the coming complexity Armageddon? What are the implications for teams and roles and the way we build and ship software? Let's talk about the industry-wide shifts underway from metrics to events, from monitoring to observability, and from caring about the system as whole to the health of each and every request.
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)Brian Brazil
If you’ve ever worried that you may have an outage someday due to your production servers not being able to handle increased user traffic, then this workshop will help put you at ease. Learn the foundations and how to apply it to your services.
Contact me at brian.brazil@robustperception.io if you'd like to learn more.
Skynet project: Monitor, analyze, scale, and maintain a system in the CloudSylvain Kalache
The goal of Skynet is to avoid human doing repetitive things and make a system doing them in a better way. System automation should be the way to go for any system management so that human can focus on stuff that really matters.
Related blog post for more informations https://engineering.linkedin.com/slideshare/skynet-project-_-monitor-scale-and-auto-heal-system-cloud
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Brian Brazil
Prometheus is a next-generation monitoring system with a time series database at it's core. Once you have a time series database, what do you do with it though? This talk will look at getting data in, and more importantly how to use the data you collect productively.
Contact us at prometheus@robustperception.io
Overview of Site Reliability Engineering (SRE) & best practicesAshutosh Agarwal
In any software organization, stability & innovation are always at loggerheads - the faster you move, the more things will break. This talk defines what SRE org looks like at high-tech organizations (Google, Uber).
Provisioning and Capacity Planning (Travel Meets Big Data)Brian Brazil
Ever worried that you’ll have an outage someday because your production servers can’t handle increased user traffic?
Then this workshop will help put you at ease! Learn the foundations and how to apply it to your services.
At the end of the workshop you will be able to:
– Estimate how much spare capacity you have in less than 5 minutes
– Estimate how much runway that capacity provides
– Determine how many servers you need
– Spot common potential problems as you scale
An Introduction to Prometheus (GrafanaCon 2016)Brian Brazil
Often what you monitor and get alerted on is defined by your tools, rather than what makes the most sense to you and your organisation. Alerts on metrics such as CPU usage which are noisy and rarely spot real problems, while outages go undetected. Monitoring systems can also be challenging to maintain, and overall provide a poor return on investment.
In the past few years several new monitoring systems have appeared with more powerful semantics and which are easier to run, which offer a way to vastly improve how your organisation operates and prepare you for a Cloud Native environment. Prometheus is one such system. This talk will look at the monitoring ideal and how whitebox monitoring with a time series database, multi-dimensional labels and a powerful querying/alerting language can free you from midnight pages.
Evolution of Monitoring and Prometheus (Dublin 2018)Brian Brazil
This talk looks at the evolution of monitoring over time, the ways in which you can approach monitoring, where Prometheus fit into all this, and how Prometheus itself has grown over time.
Monitoring What Matters: The Prometheus Approach to Whitebox Monitoring (Berl...Brian Brazil
Often what you monitor and get alerted on is defined by your tools, rather than what makes the most sense to you and your organisation. Alerts on metrics such as CPU usage which are noisy and rarely spot real problems, while outages go undetected. Monitoring systems can also be challenging to maintain, and overall provide a poor return on investment.
In the past few years several new monitoring systems have appeared with more powerful semantics and which are easier to run, which offer a way to vastly improve how your organisation operates Prometheus is one such system. This talk will look at the monitoring ideal and how whitebox monitoring with a time series database, multi-dimensional labels and a powerful querying/alerting language can free you from midnight pages.
Prometheus is a next-generation monitoring system. It lets you see you not just what your systems look like from the outside, but also gives visibility into the internals and business aspects of your systems. This allows everyone to benefit, including both operations and developers. This talk will look at the concepts behind monitoring with Prometheus, how it's designed, why it's suitable for Cloud Native environments and how you can get involved.
Observability for Emerging Infra (what got you here won't get you there)Charity Majors
Distributed systems, microservices, containers and schedulers, polyglot persistence .. modern infrastructure patterns are fluid and dynamic, chaotic and transient. So why are we still using LAMP-stack era tools to debug and monitor them? We'll cover some of the many shortcomings of traditional metrics and logs (and APM tools backed by metrics or logs), and show how complexity is their kryptonite. So how do we handle the coming complexity Armageddon? What are the implications for teams and roles and the way we build and ship software? Let's talk about the industry-wide shifts underway from metrics to events, from monitoring to observability, and from caring about the system as whole to the health of each and every request.
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)Brian Brazil
If you’ve ever worried that you may have an outage someday due to your production servers not being able to handle increased user traffic, then this workshop will help put you at ease. Learn the foundations and how to apply it to your services.
Contact me at brian.brazil@robustperception.io if you'd like to learn more.
Skynet project: Monitor, analyze, scale, and maintain a system in the CloudSylvain Kalache
The goal of Skynet is to avoid human doing repetitive things and make a system doing them in a better way. System automation should be the way to go for any system management so that human can focus on stuff that really matters.
Related blog post for more informations https://engineering.linkedin.com/slideshare/skynet-project-_-monitor-scale-and-auto-heal-system-cloud
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Brian Brazil
Prometheus is a next-generation monitoring system with a time series database at it's core. Once you have a time series database, what do you do with it though? This talk will look at getting data in, and more importantly how to use the data you collect productively.
Contact us at prometheus@robustperception.io
Overview of Site Reliability Engineering (SRE) & best practicesAshutosh Agarwal
In any software organization, stability & innovation are always at loggerheads - the faster you move, the more things will break. This talk defines what SRE org looks like at high-tech organizations (Google, Uber).
Provisioning and Capacity Planning (Travel Meets Big Data)Brian Brazil
Ever worried that you’ll have an outage someday because your production servers can’t handle increased user traffic?
Then this workshop will help put you at ease! Learn the foundations and how to apply it to your services.
At the end of the workshop you will be able to:
– Estimate how much spare capacity you have in less than 5 minutes
– Estimate how much runway that capacity provides
– Determine how many servers you need
– Spot common potential problems as you scale
Teach Your Sites to Call for Help: Automated Problem Reporting for Online Ser...Caktus Group
Keeping users happy after your site launches is easier if you know about problems before users complain. This presentation by Caktus developer Dan Poirier, delivered at PyTennessee 2018, lays out which problems you may want to be notified about and a number of tools that can help.
Angus Fletcher - Error Handling in Concurrent Systems
LOPSA East 2013 - Building a More Effective Monitoring Environment
1. Building a More Effective
Monitoring Environment
Mike Julian
Friday, May 3, 13
2. Who am I?
• Oak Ridge National Lab operations staff
• LOPSA technical staff (tech-team)
• Generalist IT engineer for ~10 years
• I am obsessed with monitoring.
Friday, May 3, 13
3. This isn’t a beginner’s talk.
Friday, May 3, 13
This isn’t a talk about how to set up Nagios or Zabbix or whatever. I’m not going to talk about why one system is better than
another. This is a talk for those who have some basic monitoring in place already and want to get more out of it.
I’m going to work on the assumption that you already some experience with monitoring systems. I’m going to try to stay as
tool-agnostic as possible, but I will be talking about some specific tools in some cases.
4. Monitoring is not a solved problem.
Friday, May 3, 13
A quick glance at Wikipedia’s entry for monitoring software shows 52 packages, and I know they’re missing many. That’s a lot of
different systems, and kinda underscores that this is an ongoing problem and not at all solved. New tools are being written every
day and there is a strong community dedicated to this problem. If you find yourself banging your head on your desk, screaming
about why your monitoring system isn’t doing what you want, rest assured: you aren’t alone.
5. There is no one-size-fits-all solution.
Friday, May 3, 13
Monitoring isn’t a single problem; it’s several problems combined under one label. There’s not a single tool out there that is
going to magically solve everything for you.
The only way you’re going to be totally happy with your monitoring system is if you write it from scratch, tailored to your
environment. That’s not feasible for most of us, but that doesn’t mean you can’t write some code to solve a problem every now
and then. Don’t be afraid to create something new: you’re going to have to, to get great monitoring. Think of it as building a
better wheel.
6. A more effective
monitoring system
• Automated
• Low noise
• Dynamically notifies
Friday, May 3, 13
Everything I’m going to talk about falls into these three categories.
7. Automated
• Config Management
• IPAM & CMDB
• Service discovery
• Self-healing
Friday, May 3, 13
I love automation. Automation appeals to my inner laziness. I’m sure you all understand.
There’s a lot of different ways to handle the automation, and different facets to it.
Since there’s so many different tools available for automation, and different monitoring systems have different levels of built-in
automation, I want to talk about the general approach.
In my current environment, we have the luxury of knowing definitively all the hosts on our network through our IPAM/CMDB
system. We also know who runs them, what team they belong to, and a few other useful details. A few SQL queries goes a long
way to automating our configuration. Rather than rely on our configuration management, I’ve chosen to rely on the CMDB, only
because that’s more central to my environment. Yours is probably different than mine, and that’s a key point: do what works for
*your* environment.
Service discovery is a bit trickier. I’ve been relying on the built-in functionality in my monitoring system to handle this for me,
but this could easily be set up from a config management system.
One thing I’ve been toying with that you might consider: self-healing. Most systems support the execution of a script when an
event occurs. For example, I have a network device that the SNMP engine likes to fall over on from time to time. When my SNMP
check fails, I could have that kick off an expect script to log into the device and restart the SNMP engine. Slightly related, you
could also have a script automatically create a ticket in your ticketing system, or toss the event info into a database. Lots of
options there.
8. • Dependencies & Parenting
• Handling redundant/HA services
• Dynamic Thresholds / Predictive
Monitoring
Low noise
Friday, May 3, 13
I hate a noisy monitoring system. It’s one of the biggest failures of any monitoring project. People get a deluge of email, most of
it false alarms, and then everyone just starts ignoring everything. It just gets worse the larger your environment is.
There’s a few ways to help fix this problem.
9. Parenting &
Dependencies
• SVR-01 depends on RACK-1-SW-01, which
depends on DC-EAST-RTR-01?
• Website depends on the SQL cluster
Friday, May 3, 13
Most modern monitoring systems support the concept of parent-child relationships. If you know something depends on
something else, then configure that.
This may not seem like a huge deal, but what if that router melts? Your monitoring system is going to suddenly dump a whole
lot of emails on you about not being able to reach every single thing in that datacenter.
Another example: if you know your website depends on SQL being up, then configure that, so you aren’t alerted twice for the
same problem. That way, if SQL goes down, you get only an alert for SQL, not an alert for both the website and SQL.
Bonus: now you can run reports that will tell you exactly what will be affected by taking down particular components of your
network.
10. Redundancy & HA
Monitoring
• Clusters
• Redundant hosts/services
Friday, May 3, 13
Let’s say you have a farm of web servers, and you know that if one dies, it’s not a show-stopper. Why, then, should you get
woken up at 3am because of it? This even applies on a small scale, such as a simple load balanced two-server setup. If you know
that one server can handle all of the traffic by itself, then it’s not worth losing sleep because the other server went down. This
applies equally well to a service cluster (such as DNS servers) as it does to host clusters (compute systems).
You can even get more complex and build some math in, for example, only alert me when 20% of all nodes are down.
This is one area where you may have to write some code.
11. Static thresholds
• Alert me when 20% of my disk is free
• Alert me when CPU utilization is at 80%
• etc, etc
Friday, May 3, 13
Problem: how big is the disk? Does 20% apply everywhere? Leaves a lot of questions unanswered...
12. Dynamic thresholds
• A spike or cliff is interesting--but what if it’s
below the static threshold?
• Holt-Winters Forecasting
• Averaging & standard deviation
• Aberrant behavior detection
• A holy grail of monitoring & a WIP
Friday, May 3, 13
I’m going to assume that you’ve already tweaked your thresholds to suit what you need.
Unfortunately, the typical threshold approach only goes so far.
Take the example of firewall connections: how do you know what a normal usage is? How do you know what an abnormal usage
is? Alerting when I hit 1000 denied connections per minute, but what if my baseline is actually 100 per minute and I suddenly
spike to 500 per minute? That’s something worth knowing. This is called aberrant behavior detection, and is a mainstay in the
security monitoring arena, but hasn’t gotten much attention in the non-security operational monitoring area.
You can write some code to run checks against data stored in rrdtool or Graphite and send results back to your monitoring
system. Or use Splunk.
14. Tools to do this
• rrdtool
• Graphite
• Splunk
• And a bit of code to tie it all together
Friday, May 3, 13
15. Dynamically notifies
Just use PagerDuty.
Seriously.
Friday, May 3, 13
I don’t mean to be a shill for them, but this is a seriously awesome service.
16. PagerDuty
• Flexible notification system
• Users create their own schedules,
including exceptions (eg, vacation)
• Email, SMS, voice
• Ability to escalate on demand or
automatically
Friday, May 3, 13
Any one who has ever configured contacts, escalations, and time periods in Nagios knows that it’s painful. PagerDuty solves this
problem by putting that configuration into a web interface, then allowing each contact to set their own notification methods,
time period exceptions (vacation, sick day, etc).
PagerDuty allows for inputs from multiple different sources, with different rules (or the same!) for each service. Anything that
can send an email or communicate over the API can be handled by PagerDuty.
One of the coolest things about PagerDuty is how it handles notifications.
17. PagerDuty
Friday, May 3, 13
The first time I got an alert, it blew my mind. Right in the text, I can acknowledge the alert, close the alert, or escalate it. So cool.
18. A few neat tools
• collectd
• Graphite
Friday, May 3, 13
I said I would try not to talk about specific tools much, but these are too valuable not to mention.
Not everything in your environment is going to fit nicely into the paradigm of a single monitoring system. A great example of
this is a guy from another division approached me about graphing input and output voltages on some R&D solar equipment. Not
exactly the typical monitoring we sysadmins do. The key aspect was that he needed it as near real-time as possible: less than
three seconds between each value. I decided for various technical and business reasons to not put them into my usual system,
but to use something different.
collectd is a poller. It’s got a bunch of plugins to collect data and then send it somewhere. In my particular case, I used collectd’s
SNMP plugin to poll the devices every second, then write the data to Graphite.
Graphite is a super cool graphing and metric storage system. It accepts data over a TCP port, in an easy format. The web
interface has a really nice looking visualization library, and has lots of different functions you can apply for some really awesome
results.
Of course, you can always run checks against Graphite data, thereby integrating things.
19. Friday, May 3, 13
This is a really simple graph of the bandwidth usage on the LOPSA production server in the last 24 hours.
Literally everything you see on this graph is customizable.
22. Thank You for Attending LOPSA-East '13
Please fill out the Trainer Evaluation
http://lopsa-east.org/2013/training-survey
Rate LOPSA-East '13
http://www.lopsa-east.org/2013/rate-lopsa-east-13
Friday, May 3, 13