SlideShare a Scribd company logo
1 of 39
Brian Brazil
Founder
What does
"Monitoring" mean?
Let's Stop Talking Past Each Other
Who am I?
Engineer passionate about running software reliably in production.
Core developer of Prometheus
Studied Computer Science in Trinity College Dublin.
Google SRE for 7 years, working on high-scale reliable systems.
Contributor to many open source projects, including Ansible, Python, Aurora
and Zookeeper.
Founder of Robust Perception, provider of commercial support and consulting
Monitoring
"Monitoring"
"Mon-i-tor-ing"
"Mon-i-tor-ing"
Do we all think of the same things when we see this word?
Historical Roots
A lot of what we do today for monitoring is based on tools and techniques that
were awesome decades ago.
Machines and services were cared for by artisan sysadmins, with loving individual
attention.
Special cases were the norm.
Historical Roots
Even though the tools have moved on greatly in both the open source and
commercial spaces, practices haven't always moved as quickly.
Often still feeding the machine with human blood.
Let's look at some of the tools of where we came from.
A little history - MRTG and RRD
In 1994 Tobias Oetiker created a perl script, which became MRTG 1.0 in 1995.
Used to graph metrics from SNMP or external programs. Stored data in constantly
rewritten ASCII file.
MRTG 2.0 moved some code to C, released 1997.
RRD started in 1997 by Tobias Oetiker to further improve performance, released
1999.
Many tools use/used RRD, e.g. Graphite and Munin.
A little history - MRTG and RRD
Source: Wikipedia
A little history - Nagios
Written initially by Ethan Galstad in 1996. MS-DOS application to do pings.
Started more properly in 1998, first release in 1999 as NetSaint (renamed in 2002
for legal reasons).
Runs scripts on a regular basis, and sends alerts based on their exit code.
Many projects inspired by/based off Nagios such as Icinga, Sensu and Zmon.
A little history - Nagios
Source: Wikipedia
Historical Heritage
These very popular tools and their offspring left us with a world where graphing is
whitebox and alerting is blackbox, and they are separate concerns.
They come from a world where machines are pets, and services tend to live on
one machine.
They come from a world where even slight deviance would be immediately
jumped upon by heroic engineers in a NOC.
We need a new perspective in a cloud native environment.
So what is monitoring?
We need a view that goes beyond alerting, graphing and jumping on every
potential problem.
We need to consider additional data sources, such as logs and browser events.
What about looking at the problem statement rather than what the tools can give
us?
What is the ultimate goal of all this "monitoring"?
Why do we monitor?
Know when things go wrong
Be able to debug and gain insight
Trending to see changes over time
Plumbing data to other systems/processes
Knowing when things go wrong
The first thing many people think of you say monitoring is alerting.
What is the wrongness we want to detect and alert on?
A blip with no real consequence, or a latency issue affecting users?
Symptoms vs Causes
Humans are limited in what they can handle.
If you alert on every single thing that might be a problem, you'll get overwhelmed
and suffer from alert fatigue.
Key problem: You care about things like user facing latency. There are hundreds
of things that could cause that.
Alerting on every possible cause is a Sisyphean task, but alerting on the symptom
of high latency is just one alert.
Example: CPU usage
Some monitoring systems don't allow you to alert on the latency of your servers.
The closest you can get is CPU usage.
False positives due to e.g. logrotate running too long.
False negatives due to deadlocks.
End result: Spammy alerts which operators learn to ignore, missing real problems.
Human attention is limited
Alerts should require intelligent human action!
Alerts should relate to actual end-user problems!
Your users don't care if a machine has a load of 4.
They care if they can't view their cat videos.
Debugging to Gain Insight
After you receive an alert notification you need to investigate it.
How do you work from a high level symptom alert such as increased latency?
You methodically drill down through your stack with dashboards to find the
subsystem that's the cause.
Break out more tools as you drill down into the suspect process.
Complementary Debugging Tools
Trending and Reporting
Alerting and debugging is short term.
Trending is medium to long term.
How is cache hit rate changing over time?
Is anyone still using that obscure feature?
When will I need more machines?
Plumbing
When all your have is a hammer, everything starts to look like a nail.
Often it'd be really convenient to use a monitoring system as a data transport as
part of some other process (often a control loop of some form).
This isn't monitoring, but it's going to happen.
If it's ~free, it's not necessarily even a bad idea.
How do we monitor?
Now knowing the three general goals of monitoring (plus plumbing), how do we go
about it?
What data do we collect? How do we process it?
What tradeoffs do we make?
Monitoring resources aren't free, so everything all the time is rarely an option.
The core is the Event
Events are what monitoring systems work off.
An event might be a HTTP request coming in, a packet being sent out, a library
call being made or a blackbox probe failing.
Events have context, such as the customer ultimately making the request, which
machines are involved, and how much data is being processed.
A single event might accumulate thousands of pieces of context as it goes through
the system, and there can be millions of events per second.
How do we monitor?
We're going to look at 4 classes of approaches to monitoring.
Profiling
Metrics
Logs
Distributed Tracing
When someone says "monitoring" they often have one of these stuck in their
head, however they're all complementary with different tradeoffs.
Profiling
Profiling is using tools like tcpdump, gdb, strace, dtrace, BPF and pretty much
anything Brendan Gregg talks about.
They give you very detailed information about individual events.
So detailed that you can't keep it all, so use must be targeted and temporary.
Great for debugging if have an idea what's wrong, not so much for anything else.
Metrics
Metrics make the tradeoff that we're going to ignore individual events, but track
how often particular contexts show up. Examples: Prometheus, Graphite.
For example you wouldn't track the exact time each HTTP request came in, but
you do track enough to tell that there were 3594 in the past minute.
And 14 of them failed. And 85 were for the login subsystem. And 5.4MB was
transferred. With 2023 cache hits.
And one of those requests hit a weird code path everyone has forgotten about.
Metric Tradeoffs
Metrics give you breadth.
You can easily track tens of thousands of metrics, but you can't break them out by
too many dimensions. E.g. breaking out metrics by email address is rarely a good
idea.
Great for figuring out what's generally going on, writing meaningful alerts,
narrowing down the scope of what needs debugging.
Not so great for tracking individual events.
Logs
Logs take the opposite approach to metrics.
They track individual events. So you can tell that Mr. Foo visited /myendpoint
yesterday at 7pm and received a 7892 byte response with status code 200.
The downside is you're limited in how many fields you can track due, likely <100.
Data volumes involved can also mean it takes some time for data to be available.
Examples: ELK stack, Graylog
Logs - But Wait There's More
Just like with how "monitoring" tends to have per-person pre-conceived ideas
there's a similar issue with "logs". Can lead to people talking past each other.
Roughly four types of logs:
Transactional, e.g. for billing or auditing. Cannot be lost.
Request, e.g. logs for individual HTTP requests or failures.
Application, e.g. I'm starting GC now, I got a SIGHUP.
Debug, e.g. individual function calls. Essentially profiling.
Each has different volume, ease of processing and durability requirements.
Distributed Tracing
Distributed tracing is really a special case of logging.
It gives each request an unique ID that is logged as the request is handled
through various services in your architecture.
It ties these logs back together to see how the request flowed through the system,
and where time was spent.
Essential for debugging large-scale distributed systems.
Examples: OpenTracing, OpenZipkin
Distributed Tracing
Source: OpenZipkin
So what does "monitoring" mean?
Not just blackbox alerts going to a NOC
Not just per-host graphs
Not just aggregated whitebox application graphs
Not just logs
Not just metrics
Not just a big TV screen on a wall
So what does "monitoring" mean?
Monitoring is the set of tools and techniques you
use to keep an eye on how your system is doing,
and keep it functional.
There is no silver bullet product you can get that does everything.
Culture and policy form part of your monitoring system too!
In Summary: Monitoring Goals
Know when things go wrong
Have alerts that require intelligent human action and directly relate to user impact
Be able to debug and gain insight
Be able to methodically go from an alert to the culprit subsystem
Trending to see changes over time
Engineering depends on data
Plumbing data to other systems/processes
It's going to happen
In Summary: Monitoring Systems
Profiling
Lots of data - too much to use it all the time.
Metrics
Great breadth - at the cost of depth.
Logs
Great depth - at the cost of breadth.
Distributed Tracing
Essential to debugging certain distributed systems problems.
Final Word
Don't be held back by what made sense 20 years ago, or what you currently do.
As we go towards a cloud native world we need to take advantage of all the new
classes of monitoring tools out there. We must scale ourselves as engineers.
Just as we no longer provision machines individually, neither should we care
about once machine being slow/down/dodgy. Care about services and end-users.
(Also, I hear Prometheus is great for metrics)
Questions?
No really, you should try Prometheus: prometheus.io
Here's a Demo: demo.robustperception.io
My Company Website: www.robustperception.io

More Related Content

What's hot

Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)Brian Brazil
 
Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)Brian Brazil
 
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)Brian Brazil
 
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)Brian Brazil
 
Ansible at FOSDEM (Ansible Dublin, 2016)
Ansible at FOSDEM (Ansible Dublin, 2016)Ansible at FOSDEM (Ansible Dublin, 2016)
Ansible at FOSDEM (Ansible Dublin, 2016)Brian Brazil
 
Prometheus design and philosophy
Prometheus design and philosophy   Prometheus design and philosophy
Prometheus design and philosophy Docker, Inc.
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Brian Brazil
 
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)Brian Brazil
 
So You Want to Write an Exporter
So You Want to Write an ExporterSo You Want to Write an Exporter
So You Want to Write an ExporterBrian Brazil
 
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)Brian Brazil
 
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Brian Brazil
 
Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...
Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...
Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...Brian Brazil
 
Prometheus Overview
Prometheus OverviewPrometheus Overview
Prometheus OverviewBrian Brazil
 
Cloud Monitoring with Prometheus
Cloud Monitoring with PrometheusCloud Monitoring with Prometheus
Cloud Monitoring with PrometheusQAware GmbH
 
Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)Brian Brazil
 
End to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenEnd to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenParis Container Day
 
The Open-Source Monitoring Landscape
The Open-Source Monitoring LandscapeThe Open-Source Monitoring Landscape
The Open-Source Monitoring LandscapeMike Merideth
 
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)Brian Brazil
 
Monitoring your Python with Prometheus (Python Ireland April 2015)
Monitoring your Python with Prometheus (Python Ireland April 2015)Monitoring your Python with Prometheus (Python Ireland April 2015)
Monitoring your Python with Prometheus (Python Ireland April 2015)Brian Brazil
 
Migrating to Prometheus: what we learned running it in production
Migrating to Prometheus: what we learned running it in productionMigrating to Prometheus: what we learned running it in production
Migrating to Prometheus: what we learned running it in productionMarco Pracucci
 

What's hot (20)

Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)Anatomy of a Prometheus Client Library (PromCon 2018)
Anatomy of a Prometheus Client Library (PromCon 2018)
 
Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)Prometheus (Prometheus London, 2016)
Prometheus (Prometheus London, 2016)
 
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
Evolving Prometheus for the Cloud Native World (FOSDEM 2018)
 
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
Staleness and Isolation in Prometheus 2.0 (PromCon 2017)
 
Ansible at FOSDEM (Ansible Dublin, 2016)
Ansible at FOSDEM (Ansible Dublin, 2016)Ansible at FOSDEM (Ansible Dublin, 2016)
Ansible at FOSDEM (Ansible Dublin, 2016)
 
Prometheus design and philosophy
Prometheus design and philosophy   Prometheus design and philosophy
Prometheus design and philosophy
 
Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)Microservices and Prometheus (Microservices NYC 2016)
Microservices and Prometheus (Microservices NYC 2016)
 
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)
Provisioning and Capacity Planning Workshop (Dogpatch Labs, September 2015)
 
So You Want to Write an Exporter
So You Want to Write an ExporterSo You Want to Write an Exporter
So You Want to Write an Exporter
 
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
Prometheus: A Next Generation Monitoring System (FOSDEM 2016)
 
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
 
Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...
Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...
Better Monitoring for Python: Inclusive Monitoring with Prometheus (Pycon Ire...
 
Prometheus Overview
Prometheus OverviewPrometheus Overview
Prometheus Overview
 
Cloud Monitoring with Prometheus
Cloud Monitoring with PrometheusCloud Monitoring with Prometheus
Cloud Monitoring with Prometheus
 
Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)Prometheus and Docker (Docker Galway, November 2015)
Prometheus and Docker (Docker Galway, November 2015)
 
End to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max IndenEnd to-end monitoring with the prometheus operator - Max Inden
End to-end monitoring with the prometheus operator - Max Inden
 
The Open-Source Monitoring Landscape
The Open-Source Monitoring LandscapeThe Open-Source Monitoring Landscape
The Open-Source Monitoring Landscape
 
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
OpenMetrics: What Does It Mean for You (PromCon 2019, Munich)
 
Monitoring your Python with Prometheus (Python Ireland April 2015)
Monitoring your Python with Prometheus (Python Ireland April 2015)Monitoring your Python with Prometheus (Python Ireland April 2015)
Monitoring your Python with Prometheus (Python Ireland April 2015)
 
Migrating to Prometheus: what we learned running it in production
Migrating to Prometheus: what we learned running it in productionMigrating to Prometheus: what we learned running it in production
Migrating to Prometheus: what we learned running it in production
 

Viewers also liked

Resume -Resume -continous monitoring
Resume -Resume -continous monitoringResume -Resume -continous monitoring
Resume -Resume -continous monitoringTony Kenny
 
Deploying services: automation with docker and ansible
Deploying services: automation with docker and ansibleDeploying services: automation with docker and ansible
Deploying services: automation with docker and ansibleJohn Zaccone
 
AtlasCamp 2015: How HipChat ships at the speed of awesome
AtlasCamp 2015: How HipChat ships at the speed of awesomeAtlasCamp 2015: How HipChat ships at the speed of awesome
AtlasCamp 2015: How HipChat ships at the speed of awesomeAtlassian
 
Roxar Multiphase Meter
Roxar Multiphase MeterRoxar Multiphase Meter
Roxar Multiphase Meterali_elkaseh
 
LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...
LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...
LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...Daniel Bryant
 
Regex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language InsteadRegex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language InsteadAll Things Open
 
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...Amazon Web Services
 
Bbc jan13 ftth_households
Bbc jan13 ftth_householdsBbc jan13 ftth_households
Bbc jan13 ftth_householdsBailey White
 
AppSphere 15 - Containers and Microservices Create New Performance Challenges
AppSphere 15 - Containers and Microservices Create New Performance ChallengesAppSphere 15 - Containers and Microservices Create New Performance Challenges
AppSphere 15 - Containers and Microservices Create New Performance ChallengesAppDynamics
 
Combining sentences with the words although and despite
Combining sentences with the words although and despiteCombining sentences with the words although and despite
Combining sentences with the words although and despiteEmily Kissner
 
Docker Swarm: Docker Native Clustering
Docker Swarm: Docker Native ClusteringDocker Swarm: Docker Native Clustering
Docker Swarm: Docker Native ClusteringDocker, Inc.
 
Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...
Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...
Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...Amazon Web Services
 
Bsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue TeamsBsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue TeamsSuraj Pratap
 
Serverless Logging with AWS Lambda and the Elastic Stack
Serverless Logging with AWS Lambda and the Elastic StackServerless Logging with AWS Lambda and the Elastic Stack
Serverless Logging with AWS Lambda and the Elastic StackEdoardo Paolo Scalafiotti
 

Viewers also liked (20)

Resume -Resume -continous monitoring
Resume -Resume -continous monitoringResume -Resume -continous monitoring
Resume -Resume -continous monitoring
 
Deploying services: automation with docker and ansible
Deploying services: automation with docker and ansibleDeploying services: automation with docker and ansible
Deploying services: automation with docker and ansible
 
AtlasCamp 2015: How HipChat ships at the speed of awesome
AtlasCamp 2015: How HipChat ships at the speed of awesomeAtlasCamp 2015: How HipChat ships at the speed of awesome
AtlasCamp 2015: How HipChat ships at the speed of awesome
 
114 Numalliance
114 Numalliance114 Numalliance
114 Numalliance
 
Roxar Multiphase Meter
Roxar Multiphase MeterRoxar Multiphase Meter
Roxar Multiphase Meter
 
LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...
LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...
LJC Mashup "Building Java Microservices for the Cloud && Chuck Norris Doesn't...
 
Regex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language InsteadRegex Considered Harmful: Use Rosie Pattern Language Instead
Regex Considered Harmful: Use Rosie Pattern Language Instead
 
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
AWS May Webinar Series - Streaming Data Processing with Amazon Kinesis and AW...
 
Bbc jan13 ftth_households
Bbc jan13 ftth_householdsBbc jan13 ftth_households
Bbc jan13 ftth_households
 
TrendsByte Presentation
TrendsByte PresentationTrendsByte Presentation
TrendsByte Presentation
 
AppSphere 15 - Containers and Microservices Create New Performance Challenges
AppSphere 15 - Containers and Microservices Create New Performance ChallengesAppSphere 15 - Containers and Microservices Create New Performance Challenges
AppSphere 15 - Containers and Microservices Create New Performance Challenges
 
EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...
EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...
EVOLVE'16 | Enhance | Anil Kalbag & Anshul Chhabra | Comparative Architecture...
 
Combining sentences with the words although and despite
Combining sentences with the words although and despiteCombining sentences with the words although and despite
Combining sentences with the words although and despite
 
Automating interactions with Zabbix (Raymond Kuiper / 12-02-2015)
Automating interactions with Zabbix (Raymond Kuiper / 12-02-2015)Automating interactions with Zabbix (Raymond Kuiper / 12-02-2015)
Automating interactions with Zabbix (Raymond Kuiper / 12-02-2015)
 
AWS Cost Visualizer
AWS Cost VisualizerAWS Cost Visualizer
AWS Cost Visualizer
 
Docker Swarm: Docker Native Clustering
Docker Swarm: Docker Native ClusteringDocker Swarm: Docker Native Clustering
Docker Swarm: Docker Native Clustering
 
EVOLVE'16 | Enhance | Gordon Pike | Rev Up Your Marketing Engine
EVOLVE'16 | Enhance | Gordon Pike | Rev Up Your Marketing EngineEVOLVE'16 | Enhance | Gordon Pike | Rev Up Your Marketing Engine
EVOLVE'16 | Enhance | Gordon Pike | Rev Up Your Marketing Engine
 
Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...
Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...
Visualize your data in Data Lake with AWS Athena and AWS Quicksight Hands-on ...
 
Bsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue TeamsBsides Delhi Security Automation for Red and Blue Teams
Bsides Delhi Security Automation for Red and Blue Teams
 
Serverless Logging with AWS Lambda and the Elastic Stack
Serverless Logging with AWS Lambda and the Elastic StackServerless Logging with AWS Lambda and the Elastic Stack
Serverless Logging with AWS Lambda and the Elastic Stack
 

Similar to What does "monitoring" mean? (FOSDEM 2017)

Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisAnton Chuvakin
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)Brian Brazil
 
Chaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just ChaosChaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just ChaosCharity Majors
 
Audit logs for Security and Compliance
Audit logs for Security and ComplianceAudit logs for Security and Compliance
Audit logs for Security and ComplianceAnton Chuvakin
 
The tops for collecting network based evidenceyou think that your.pdf
The tops for collecting network based evidenceyou think that your.pdfThe tops for collecting network based evidenceyou think that your.pdf
The tops for collecting network based evidenceyou think that your.pdfnoelbuddy
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMAAnton Chuvakin
 
Analytics Driven SIEM Workshop
Analytics Driven SIEM WorkshopAnalytics Driven SIEM Workshop
Analytics Driven SIEM WorkshopSplunk
 
Red lambda FAQ's
Red lambda FAQ'sRed lambda FAQ's
Red lambda FAQ'sIla Group
 
20 Trip-Wire-.pdf
20 Trip-Wire-.pdf20 Trip-Wire-.pdf
20 Trip-Wire-.pdfG Srinu
 
20 Trip-Wire-.pdf
20 Trip-Wire-.pdf20 Trip-Wire-.pdf
20 Trip-Wire-.pdfG Srinu
 
Advanced malware analysis training session3 botnet analysis part2
Advanced malware analysis training session3 botnet analysis part2Advanced malware analysis training session3 botnet analysis part2
Advanced malware analysis training session3 botnet analysis part2Cysinfo Cyber Security Community
 
SplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud DetectionSplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud DetectionSplunk
 
Observability für alle
Observability für alleObservability für alle
Observability für alleQAware GmbH
 
Are you ready for the next attack? Reviewing the SP Security Checklist
Are you ready for the next attack? Reviewing the SP Security ChecklistAre you ready for the next attack? Reviewing the SP Security Checklist
Are you ready for the next attack? Reviewing the SP Security ChecklistAPNIC
 
Are you ready for the next attack? reviewing the sp security checklist (apnic...
Are you ready for the next attack? reviewing the sp security checklist (apnic...Are you ready for the next attack? reviewing the sp security checklist (apnic...
Are you ready for the next attack? reviewing the sp security checklist (apnic...Barry Greene
 
Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Skynet project: Monitor, analyze, scale, and maintain a system in the CloudSkynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Skynet project: Monitor, analyze, scale, and maintain a system in the CloudSylvain Kalache
 
Logging "BrainBox" Short Article
Logging "BrainBox" Short ArticleLogging "BrainBox" Short Article
Logging "BrainBox" Short ArticleAnton Chuvakin
 
Splunk for ITOA Breakout Session
Splunk for ITOA Breakout SessionSplunk for ITOA Breakout Session
Splunk for ITOA Breakout SessionSplunk
 
Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Anton Chuvakin
 

Similar to What does "monitoring" mean? (FOSDEM 2017) (20)

Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log Analysis
 
An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
 
Chaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just ChaosChaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just Chaos
 
Audit logs for Security and Compliance
Audit logs for Security and ComplianceAudit logs for Security and Compliance
Audit logs for Security and Compliance
 
The tops for collecting network based evidenceyou think that your.pdf
The tops for collecting network based evidenceyou think that your.pdfThe tops for collecting network based evidenceyou think that your.pdf
The tops for collecting network based evidenceyou think that your.pdf
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMA
 
ambient-computing
ambient-computingambient-computing
ambient-computing
 
Analytics Driven SIEM Workshop
Analytics Driven SIEM WorkshopAnalytics Driven SIEM Workshop
Analytics Driven SIEM Workshop
 
Red lambda FAQ's
Red lambda FAQ'sRed lambda FAQ's
Red lambda FAQ's
 
20 Trip-Wire-.pdf
20 Trip-Wire-.pdf20 Trip-Wire-.pdf
20 Trip-Wire-.pdf
 
20 Trip-Wire-.pdf
20 Trip-Wire-.pdf20 Trip-Wire-.pdf
20 Trip-Wire-.pdf
 
Advanced malware analysis training session3 botnet analysis part2
Advanced malware analysis training session3 botnet analysis part2Advanced malware analysis training session3 botnet analysis part2
Advanced malware analysis training session3 botnet analysis part2
 
SplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud DetectionSplunkLive! Splunk for Insider Threats and Fraud Detection
SplunkLive! Splunk for Insider Threats and Fraud Detection
 
Observability für alle
Observability für alleObservability für alle
Observability für alle
 
Are you ready for the next attack? Reviewing the SP Security Checklist
Are you ready for the next attack? Reviewing the SP Security ChecklistAre you ready for the next attack? Reviewing the SP Security Checklist
Are you ready for the next attack? Reviewing the SP Security Checklist
 
Are you ready for the next attack? reviewing the sp security checklist (apnic...
Are you ready for the next attack? reviewing the sp security checklist (apnic...Are you ready for the next attack? reviewing the sp security checklist (apnic...
Are you ready for the next attack? reviewing the sp security checklist (apnic...
 
Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Skynet project: Monitor, analyze, scale, and maintain a system in the CloudSkynet project: Monitor, analyze, scale, and maintain a system in the Cloud
Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud
 
Logging "BrainBox" Short Article
Logging "BrainBox" Short ArticleLogging "BrainBox" Short Article
Logging "BrainBox" Short Article
 
Splunk for ITOA Breakout Session
Splunk for ITOA Breakout SessionSplunk for ITOA Breakout Session
Splunk for ITOA Breakout Session
 
Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008
 

More from Brian Brazil

Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)Brian Brazil
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Brian Brazil
 
Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)Brian Brazil
 
An Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQLAn Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQLBrian Brazil
 
Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)Brian Brazil
 
Prometheus (Monitorama 2016)
Prometheus (Monitorama 2016)Prometheus (Monitorama 2016)
Prometheus (Monitorama 2016)Brian Brazil
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Brian Brazil
 
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Brian Brazil
 
Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)Brian Brazil
 

More from Brian Brazil (9)

Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
Evaluating Prometheus Knowledge in Interviews (PromCon 2018)
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)
 
Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)Rule 110 for Prometheus (PromCon 2017)
Rule 110 for Prometheus (PromCon 2017)
 
An Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQLAn Exploration of the Formal Properties of PromQL
An Exploration of the Formal Properties of PromQL
 
Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)Life of a Label (PromCon2016, Berlin)
Life of a Label (PromCon2016, Berlin)
 
Prometheus (Monitorama 2016)
Prometheus (Monitorama 2016)Prometheus (Monitorama 2016)
Prometheus (Monitorama 2016)
 
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
Your data is in Prometheus, now what? (CurrencyFair Engineering Meetup, 2016)
 
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)
 
Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)Prometheus (Microsoft, 2016)
Prometheus (Microsoft, 2016)
 

Recently uploaded

Contact Rya Baby for Call Girls New Delhi
Contact Rya Baby for Call Girls New DelhiContact Rya Baby for Call Girls New Delhi
Contact Rya Baby for Call Girls New Delhimiss dipika
 
PHP-based rendering of TYPO3 Documentation
PHP-based rendering of TYPO3 DocumentationPHP-based rendering of TYPO3 Documentation
PHP-based rendering of TYPO3 DocumentationLinaWolf1
 
Top 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxTop 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxDyna Gilbert
 
A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)
A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)
A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)Christopher H Felton
 
Film cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasaFilm cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasa494f574xmv
 
Font Performance - NYC WebPerf Meetup April '24
Font Performance - NYC WebPerf Meetup April '24Font Performance - NYC WebPerf Meetup April '24
Font Performance - NYC WebPerf Meetup April '24Paul Calvano
 
NSX-T and Service Interfaces presentation
NSX-T and Service Interfaces presentationNSX-T and Service Interfaces presentation
NSX-T and Service Interfaces presentationMarko4394
 
SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predieusebiomeyer
 
定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一
定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一
定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一Fs
 
Magic exist by Marta Loveguard - presentation.pptx
Magic exist by Marta Loveguard - presentation.pptxMagic exist by Marta Loveguard - presentation.pptx
Magic exist by Marta Loveguard - presentation.pptxMartaLoveguard
 
Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170
Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170
Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170Sonam Pathan
 
Blepharitis inflammation of eyelid symptoms cause everything included along w...
Blepharitis inflammation of eyelid symptoms cause everything included along w...Blepharitis inflammation of eyelid symptoms cause everything included along w...
Blepharitis inflammation of eyelid symptoms cause everything included along w...Excelmac1
 
定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一
定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一
定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一Fs
 
Q4-1-Illustrating-Hypothesis-Testing.pptx
Q4-1-Illustrating-Hypothesis-Testing.pptxQ4-1-Illustrating-Hypothesis-Testing.pptx
Q4-1-Illustrating-Hypothesis-Testing.pptxeditsforyah
 
Git and Github workshop GDSC MLRITM
Git and Github  workshop GDSC MLRITMGit and Github  workshop GDSC MLRITM
Git and Github workshop GDSC MLRITMgdsc13
 
Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作
Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作
Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作ys8omjxb
 
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书zdzoqco
 
办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一
办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一
办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一z xss
 

Recently uploaded (20)

Contact Rya Baby for Call Girls New Delhi
Contact Rya Baby for Call Girls New DelhiContact Rya Baby for Call Girls New Delhi
Contact Rya Baby for Call Girls New Delhi
 
PHP-based rendering of TYPO3 Documentation
PHP-based rendering of TYPO3 DocumentationPHP-based rendering of TYPO3 Documentation
PHP-based rendering of TYPO3 Documentation
 
young call girls in Uttam Nagar🔝 9953056974 🔝 Delhi escort Service
young call girls in Uttam Nagar🔝 9953056974 🔝 Delhi escort Serviceyoung call girls in Uttam Nagar🔝 9953056974 🔝 Delhi escort Service
young call girls in Uttam Nagar🔝 9953056974 🔝 Delhi escort Service
 
Hot Sexy call girls in Rk Puram 🔝 9953056974 🔝 Delhi escort Service
Hot Sexy call girls in  Rk Puram 🔝 9953056974 🔝 Delhi escort ServiceHot Sexy call girls in  Rk Puram 🔝 9953056974 🔝 Delhi escort Service
Hot Sexy call girls in Rk Puram 🔝 9953056974 🔝 Delhi escort Service
 
Top 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptxTop 10 Interactive Website Design Trends in 2024.pptx
Top 10 Interactive Website Design Trends in 2024.pptx
 
A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)
A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)
A Good Girl's Guide to Murder (A Good Girl's Guide to Murder, #1)
 
Film cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasaFilm cover research (1).pptxsdasdasdasdasdasa
Film cover research (1).pptxsdasdasdasdasdasa
 
Font Performance - NYC WebPerf Meetup April '24
Font Performance - NYC WebPerf Meetup April '24Font Performance - NYC WebPerf Meetup April '24
Font Performance - NYC WebPerf Meetup April '24
 
NSX-T and Service Interfaces presentation
NSX-T and Service Interfaces presentationNSX-T and Service Interfaces presentation
NSX-T and Service Interfaces presentation
 
SCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is prediSCM Symposium PPT Format Customer loyalty is predi
SCM Symposium PPT Format Customer loyalty is predi
 
定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一
定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一
定制(AUT毕业证书)新西兰奥克兰理工大学毕业证成绩单原版一比一
 
Magic exist by Marta Loveguard - presentation.pptx
Magic exist by Marta Loveguard - presentation.pptxMagic exist by Marta Loveguard - presentation.pptx
Magic exist by Marta Loveguard - presentation.pptx
 
Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170
Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170
Call Girls In The Ocean Pearl Retreat Hotel New Delhi 9873777170
 
Blepharitis inflammation of eyelid symptoms cause everything included along w...
Blepharitis inflammation of eyelid symptoms cause everything included along w...Blepharitis inflammation of eyelid symptoms cause everything included along w...
Blepharitis inflammation of eyelid symptoms cause everything included along w...
 
定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一
定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一
定制(Lincoln毕业证书)新西兰林肯大学毕业证成绩单原版一比一
 
Q4-1-Illustrating-Hypothesis-Testing.pptx
Q4-1-Illustrating-Hypothesis-Testing.pptxQ4-1-Illustrating-Hypothesis-Testing.pptx
Q4-1-Illustrating-Hypothesis-Testing.pptx
 
Git and Github workshop GDSC MLRITM
Git and Github  workshop GDSC MLRITMGit and Github  workshop GDSC MLRITM
Git and Github workshop GDSC MLRITM
 
Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作
Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作
Potsdam FH学位证,波茨坦应用技术大学毕业证书1:1制作
 
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
办理多伦多大学毕业证成绩单|购买加拿大UTSG文凭证书
 
办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一
办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一
办理(UofR毕业证书)罗切斯特大学毕业证成绩单原版一比一
 

What does "monitoring" mean? (FOSDEM 2017)

  • 1. Brian Brazil Founder What does "Monitoring" mean? Let's Stop Talking Past Each Other
  • 2. Who am I? Engineer passionate about running software reliably in production. Core developer of Prometheus Studied Computer Science in Trinity College Dublin. Google SRE for 7 years, working on high-scale reliable systems. Contributor to many open source projects, including Ansible, Python, Aurora and Zookeeper. Founder of Robust Perception, provider of commercial support and consulting
  • 6. "Mon-i-tor-ing" Do we all think of the same things when we see this word?
  • 7. Historical Roots A lot of what we do today for monitoring is based on tools and techniques that were awesome decades ago. Machines and services were cared for by artisan sysadmins, with loving individual attention. Special cases were the norm.
  • 8. Historical Roots Even though the tools have moved on greatly in both the open source and commercial spaces, practices haven't always moved as quickly. Often still feeding the machine with human blood. Let's look at some of the tools of where we came from.
  • 9. A little history - MRTG and RRD In 1994 Tobias Oetiker created a perl script, which became MRTG 1.0 in 1995. Used to graph metrics from SNMP or external programs. Stored data in constantly rewritten ASCII file. MRTG 2.0 moved some code to C, released 1997. RRD started in 1997 by Tobias Oetiker to further improve performance, released 1999. Many tools use/used RRD, e.g. Graphite and Munin.
  • 10. A little history - MRTG and RRD Source: Wikipedia
  • 11. A little history - Nagios Written initially by Ethan Galstad in 1996. MS-DOS application to do pings. Started more properly in 1998, first release in 1999 as NetSaint (renamed in 2002 for legal reasons). Runs scripts on a regular basis, and sends alerts based on their exit code. Many projects inspired by/based off Nagios such as Icinga, Sensu and Zmon.
  • 12. A little history - Nagios Source: Wikipedia
  • 13. Historical Heritage These very popular tools and their offspring left us with a world where graphing is whitebox and alerting is blackbox, and they are separate concerns. They come from a world where machines are pets, and services tend to live on one machine. They come from a world where even slight deviance would be immediately jumped upon by heroic engineers in a NOC. We need a new perspective in a cloud native environment.
  • 14. So what is monitoring? We need a view that goes beyond alerting, graphing and jumping on every potential problem. We need to consider additional data sources, such as logs and browser events. What about looking at the problem statement rather than what the tools can give us? What is the ultimate goal of all this "monitoring"?
  • 15. Why do we monitor? Know when things go wrong Be able to debug and gain insight Trending to see changes over time Plumbing data to other systems/processes
  • 16. Knowing when things go wrong The first thing many people think of you say monitoring is alerting. What is the wrongness we want to detect and alert on? A blip with no real consequence, or a latency issue affecting users?
  • 17. Symptoms vs Causes Humans are limited in what they can handle. If you alert on every single thing that might be a problem, you'll get overwhelmed and suffer from alert fatigue. Key problem: You care about things like user facing latency. There are hundreds of things that could cause that. Alerting on every possible cause is a Sisyphean task, but alerting on the symptom of high latency is just one alert.
  • 18. Example: CPU usage Some monitoring systems don't allow you to alert on the latency of your servers. The closest you can get is CPU usage. False positives due to e.g. logrotate running too long. False negatives due to deadlocks. End result: Spammy alerts which operators learn to ignore, missing real problems.
  • 19. Human attention is limited Alerts should require intelligent human action! Alerts should relate to actual end-user problems! Your users don't care if a machine has a load of 4. They care if they can't view their cat videos.
  • 20. Debugging to Gain Insight After you receive an alert notification you need to investigate it. How do you work from a high level symptom alert such as increased latency? You methodically drill down through your stack with dashboards to find the subsystem that's the cause. Break out more tools as you drill down into the suspect process.
  • 22. Trending and Reporting Alerting and debugging is short term. Trending is medium to long term. How is cache hit rate changing over time? Is anyone still using that obscure feature? When will I need more machines?
  • 23. Plumbing When all your have is a hammer, everything starts to look like a nail. Often it'd be really convenient to use a monitoring system as a data transport as part of some other process (often a control loop of some form). This isn't monitoring, but it's going to happen. If it's ~free, it's not necessarily even a bad idea.
  • 24. How do we monitor? Now knowing the three general goals of monitoring (plus plumbing), how do we go about it? What data do we collect? How do we process it? What tradeoffs do we make? Monitoring resources aren't free, so everything all the time is rarely an option.
  • 25. The core is the Event Events are what monitoring systems work off. An event might be a HTTP request coming in, a packet being sent out, a library call being made or a blackbox probe failing. Events have context, such as the customer ultimately making the request, which machines are involved, and how much data is being processed. A single event might accumulate thousands of pieces of context as it goes through the system, and there can be millions of events per second.
  • 26. How do we monitor? We're going to look at 4 classes of approaches to monitoring. Profiling Metrics Logs Distributed Tracing When someone says "monitoring" they often have one of these stuck in their head, however they're all complementary with different tradeoffs.
  • 27. Profiling Profiling is using tools like tcpdump, gdb, strace, dtrace, BPF and pretty much anything Brendan Gregg talks about. They give you very detailed information about individual events. So detailed that you can't keep it all, so use must be targeted and temporary. Great for debugging if have an idea what's wrong, not so much for anything else.
  • 28. Metrics Metrics make the tradeoff that we're going to ignore individual events, but track how often particular contexts show up. Examples: Prometheus, Graphite. For example you wouldn't track the exact time each HTTP request came in, but you do track enough to tell that there were 3594 in the past minute. And 14 of them failed. And 85 were for the login subsystem. And 5.4MB was transferred. With 2023 cache hits. And one of those requests hit a weird code path everyone has forgotten about.
  • 29. Metric Tradeoffs Metrics give you breadth. You can easily track tens of thousands of metrics, but you can't break them out by too many dimensions. E.g. breaking out metrics by email address is rarely a good idea. Great for figuring out what's generally going on, writing meaningful alerts, narrowing down the scope of what needs debugging. Not so great for tracking individual events.
  • 30. Logs Logs take the opposite approach to metrics. They track individual events. So you can tell that Mr. Foo visited /myendpoint yesterday at 7pm and received a 7892 byte response with status code 200. The downside is you're limited in how many fields you can track due, likely <100. Data volumes involved can also mean it takes some time for data to be available. Examples: ELK stack, Graylog
  • 31. Logs - But Wait There's More Just like with how "monitoring" tends to have per-person pre-conceived ideas there's a similar issue with "logs". Can lead to people talking past each other. Roughly four types of logs: Transactional, e.g. for billing or auditing. Cannot be lost. Request, e.g. logs for individual HTTP requests or failures. Application, e.g. I'm starting GC now, I got a SIGHUP. Debug, e.g. individual function calls. Essentially profiling. Each has different volume, ease of processing and durability requirements.
  • 32. Distributed Tracing Distributed tracing is really a special case of logging. It gives each request an unique ID that is logged as the request is handled through various services in your architecture. It ties these logs back together to see how the request flowed through the system, and where time was spent. Essential for debugging large-scale distributed systems. Examples: OpenTracing, OpenZipkin
  • 34. So what does "monitoring" mean? Not just blackbox alerts going to a NOC Not just per-host graphs Not just aggregated whitebox application graphs Not just logs Not just metrics Not just a big TV screen on a wall
  • 35. So what does "monitoring" mean? Monitoring is the set of tools and techniques you use to keep an eye on how your system is doing, and keep it functional. There is no silver bullet product you can get that does everything. Culture and policy form part of your monitoring system too!
  • 36. In Summary: Monitoring Goals Know when things go wrong Have alerts that require intelligent human action and directly relate to user impact Be able to debug and gain insight Be able to methodically go from an alert to the culprit subsystem Trending to see changes over time Engineering depends on data Plumbing data to other systems/processes It's going to happen
  • 37. In Summary: Monitoring Systems Profiling Lots of data - too much to use it all the time. Metrics Great breadth - at the cost of depth. Logs Great depth - at the cost of breadth. Distributed Tracing Essential to debugging certain distributed systems problems.
  • 38. Final Word Don't be held back by what made sense 20 years ago, or what you currently do. As we go towards a cloud native world we need to take advantage of all the new classes of monitoring tools out there. We must scale ourselves as engineers. Just as we no longer provision machines individually, neither should we care about once machine being slow/down/dodgy. Care about services and end-users. (Also, I hear Prometheus is great for metrics)
  • 39. Questions? No really, you should try Prometheus: prometheus.io Here's a Demo: demo.robustperception.io My Company Website: www.robustperception.io