Monitoring and metrics in the cloud

•

0 likes•1,025 views

The document discusses metrics and monitoring in the cloud. It provides examples of metrics and monitoring tools used over time as systems grew from a small data center to a large cloud environment with hundreds of servers. It notes the challenges of using tools like Nagios at large scale and how new tools like Sensu are designed specifically for dynamic cloud environments.

Technology

Metrics and Monitoring in the cloud

David Lutz

@dlutzy

The objective of metrics is to
make pretty graphs…

The objective of metrics is to
make pretty graphs...
in order to understand the performance
and capacity
of your systems and how they vary over time.

The objective of monitoring is to…
make the Operations-guy-on-call’s life hell.

The objective of monitoring is to
check that the system is working as expected
and take action if some component isn't.

“Those who cannot remember the past are
condemned to repeat it” - George Santayana

So here’s a case study…

A long time ago in a data centre far,
far away….

Complete system includes humans to run it!
Human Factors Engineering.
http://en.wikipedia.org/wiki/Human_factors

2 x Linux Engineers
1 x Network Engineer
1 x Do Anything Guy
1 x Developer

No Monitoring or Metrics.

Black Box. Completely blind.

Large Development team
External Consultants
ITIL Process people

5 x Linux Engineers
1 x Network Engineer
2 x Database Administrators
and
Part of an Infrastructure team that included
Virtualization specialists
Storage specialists
Hardware specialists

WTF happened? It grew…

Virtualization / Cloud

Cloud / Virtualization

Approximately 400 servers

Still using Nagios and Cacti

15 minutes to add server manually.
1 hour or more to add a new check.

And Ganglia.
And External SAAS tools:
New Relic. Gomez. Omniture.

What’s different about the cloud?

• Servers come and go
• Sometimes automatically with auto-scaling
• Topologies and Architectures change rapidly
• Driven from Configuration Management
Systems

The problems with Nagios

• Clunky UI.
• Monolithic design.
• Hard to scale.
• Hard to add nodes
dynamically.

sensu-server
sensu-client
sensu-api
sensu-dashboard

• JSON everywhere
• Can re-use Nagios checks
• Messaging oriented architecture
• Designed to be driven from
Config Management tools
• Supports dynamic topologies

David Lutz

99designs

@dlutzy

meetup.com/Infrastructure-Coders/

To defend against attacks, think like a hacker. But does that mean you need to be a DevOps expert? Security researchers today need to discover new attack techniques. However, much of their focus is diverged to backend coding. We share how to build an infrastructure for researchers that allows them concentrate on business logic and writing hacker “tasks”. Using Docker and Kubernetes on Google Cloud, these tasks can then be performed in parallel and without a lot of DevOps hassle. Our technique removes two common barriers: first, long and risky deployment processes and second, low transparency within the production system. Promise to share the stupid things too.

Tracing Summit 2014, Düsseldorf. What can Linux learn from DTrace: what went well, and what didn't go well, on its path to success? This talk will discuss not just the DTrace software, but lessons from the marketing and adoption of a system tracer, and an inside look at how DTrace was really deployed and used in production environments. It will also cover ongoing problems with DTrace, and how Linux may surpass them and continue to advance the field of system tracing. A world expert and core contributor to DTrace, Brendan now works at Netflix on Linux performance with the various Linux tracers (ftrace, perf_events, eBPF, SystemTap, ktap, sysdig, LTTng, and the DTrace Linux ports), and will summarize his experiences and suggestions for improvements. He has also been contributing to various tracers: recently promoting ftrace and perf_events adoption through articles and front-end scripts, and testing eBPF.

POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...

Liming Zhu

Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...

Demi Ben-Ari

Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...

Codemotion

Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system using tools like: Web Services,Spark,Cassandra,MongoDB,AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.

Lecture 12 monitoring the networkWiliam Ferraciolli

Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...

Codemotion

Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk, we’ll mention all of the aspects that you should take into consideration when monitoring a distributed system using tools like Web Services, Spark, Cassandra, MongoDB, AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.

Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...

Demi Ben-Ari

Which watcher watches CloudWatch

David Lutz

Spaghetti devops

Alessandro Franceschi

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

Sylvain Kalache

QCon SF 2017 - Microservices: Service-Oriented Development

Ambassador Labs

Conventional wisdom is that microservices is an architecture that is the spiritual successor to service-oriented architecture. While true, this myopic view of microservices ignores some of the profound workflow shifts in today’s microservices organizations. The reality is that microservices is an architecture _and_ workflow. In this talk, we’ll introduce the workflow of service-oriented development. Rafael will talk about how the real goal of microservices is to break up a monolithic development workflow. We’ll show you how, by breaking up your workflow, you can build software that lets you move fast and make things.

Architectural Decisions: Smoothly and Consistently

Comsysto Reply GmbH

Abstract The idea of this talk is to help development teams to make correct architectural decisions. Andrei will highlight the basic architectural principles and show ways to achieve architecture that is good enough to cover the project requirements and evolve in the future. He will also present several cases from real projects, where wrong, missing, or over-sophisticated architecture decisions really hurt the development teams: - Painful sharing: do shared modules increase reusability or will be the source of problems? - Microservices are the solution to every problem! - Non-extensible extensibility: too sophisticated configuration hurts - Over fine-grained: incorrect splitting to Microservices can make life even harder as with monolith - Convey horizontal split: how organizational driven split can jeopardise the architecture - Model-driven: central responsibility blocks and limits the team - Cargo cult: blindly following patterns and rule can produce an unmaintainable system - Freestyle architecture: what happens if teams completely ignore architecture - Improve with less intelligence: smart endpoint and dumb pipes

Architectural Decisions: Smoothly and Consistently

Comsysto Reply GmbH

Thinking DevOps in the era of the Cloud - Demi Ben-Ari

Demi Ben-Ari

The lines between Development and Operations people have gotten blurry and lots of skills needs to be held by both sides. In the talk we'll talk about all of the considerations that are needed to be taken when creating a development and production environment, mentioning Continuous Integration, Continuous Deployment and the Buzzword "DevOps", also talking about some real implementations in the industry. Of course how can we leave out the real enabler of the whole deal, "The Cloud", Giving us a tool set that makes life much easier when implementing all of these practices.

Monitoring Big Data Systems - "The Simple Way"

Demi Ben-Ari

Once you start working with distributed Big Data systems, you start discovering a whole bunch of problems you won’t find in monolithic systems. All of a sudden to monitor all of the components becomes a big data problem itself. In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system once you’re using tools like: Web Services, Apache Spark, Cassandra, MongoDB, Amazon Web Services. Not only the tools, what should you monitor about the actual data that flows in the system? And we’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy. Demi Ben-Ari is a Co-Founder and CTO @ Panorays. Demi has over 9 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems. Describing himself as a software development groupie, Interested in tackling cutting edge technologies. Demi is also a co-founder of the “Big Things” Big Data community: http://somebigthings.com/big-things-intro/

Continuous Delivery Amsterdam - Microservices in action at the Dutch National...

Bert Jan Schrijver

Machine Learning Infrastructure

SigOpt

As data science workloads grow, so does their need for infrastructure. But, is it fair to ask data scientists to also become infrastructure experts? If not the data scientists, then, who is responsible for spinning up and managing data science infrastructure? This talk will address the context in which ML infrastructure is emerging, walk through two examples of ML infrastructure tools for launching hyperparameter optimization jobs, and end with some thoughts for building better tools in the future. Originally given as a talk at the PyData Ann Arbor meetup (https://www.meetup.com/PyData-Ann-Arbor/events/260380989/)

Scaling Pinterest's Monitoring

Brian Overstreet

2014 devops conferences

David Lutz

antifragility - DevOps Melbourne November 2013

David Lutz

Similar to Monitoring and metrics in the cloud

Webinar: Code Faster on Kubernetes

Ambassador Labs

Infrastructure - a journey from datacentres to cloud

Equal Experts

From DTrace to Linux

Brendan Gregg

POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...

Liming Zhu

Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...

Demi Ben-Ari

Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...

Codemotion

Lecture 12 monitoring the networkWiliam Ferraciolli

Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...

Codemotion

Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk, we’ll mention all of the aspects that you should take into consideration when monitoring a distributed system using tools like Web Services, Spark, Cassandra, MongoDB, AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.

Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...

Demi Ben-Ari

Which watcher watches CloudWatch

David Lutz

Spaghetti devops

Alessandro Franceschi

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

Sylvain Kalache

QCon SF 2017 - Microservices: Service-Oriented Development

Ambassador Labs

Architectural Decisions: Smoothly and Consistently

Comsysto Reply GmbH

Architectural Decisions: Smoothly and Consistently

Comsysto Reply GmbH

Thinking DevOps in the era of the Cloud - Demi Ben-Ari

Demi Ben-Ari

Monitoring Big Data Systems - "The Simple Way"

Demi Ben-Ari

Continuous Delivery Amsterdam - Microservices in action at the Dutch National...

Bert Jan Schrijver

Machine Learning Infrastructure

SigOpt

Scaling Pinterest's Monitoring

Brian Overstreet

Similar to Monitoring and metrics in the cloud (20)

Webinar: Code Faster on Kubernetes

Infrastructure - a journey from datacentres to cloud

From DTrace to Linux

POD-Diagnosis: Error Detection and Diagnosis of Sporadic Operations on Cloud ...

Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...

Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...

Lecture 12 monitoring the network

Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...

Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...

Which watcher watches CloudWatch

Spaghetti devops

Skynet project: Monitor, analyze, scale, and maintain a system in the Cloud

QCon SF 2017 - Microservices: Service-Oriented Development

Architectural Decisions: Smoothly and Consistently

Thinking DevOps in the era of the Cloud - Demi Ben-Ari

Monitoring Big Data Systems - "The Simple Way"

Continuous Delivery Amsterdam - Microservices in action at the Dutch National...

Machine Learning Infrastructure

Scaling Pinterest's Monitoring

Recently uploaded

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

The Future of Platform Engineering

Jemma Hussein Allen

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Bits & Pixels using AI for Good.........

Alison B. Lowndes

JMeter webinar - integration with InfluxDB and Grafana

RTTS

Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application. In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics. Length: 30 minutes Session Overview ------------------------------------------- During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana: - What out-of-the-box solutions are available for real-time monitoring JMeter tests? - What are the benefits of integrating InfluxDB and Grafana into the load testing stack? - Which features are provided by Grafana? - Demonstration of InfluxDB and Grafana using a practice web application To view the webinar recording, go to: https://www.rttsweb.com/jmeter-integration-webinar

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

Knowledge engineering: from people to machines and back

Elena Simperl

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Product School

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

Recently uploaded (20)

Mission to Decommission: Importance of Decommissioning Products to Increase E...

The Art of the Pitch: WordPress Relationships and Sales

The Future of Platform Engineering

PCI PIN Basics Webinar from the Controlcase Team

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Connector Corner: Automate dynamic content and events by pushing a button

Bits & Pixels using AI for Good.........

JMeter webinar - integration with InfluxDB and Grafana

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Monitoring Java Application Security with JDK Tools and JFR Events

Knowledge engineering: from people to machines and back

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...

Essentials of Automations: Optimizing FME Workflows with Parameters

Key Trends Shaping the Future of Infrastructure.pdf

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Neuro-symbolic is not enough, we need neuro-*semantic*

Monitoring and metrics in the cloud

1. Metrics and Monitoring in the cloud David Lutz @dlutzy

2. The objective of metrics is to make pretty graphs…

3. The objective of metrics is to make pretty graphs... in order to understand the performance and capacity of your systems and how they vary over time.

4. The objective of monitoring is to… make the Operations-guy-on-call’s life hell.

5. The objective of monitoring is to check that the system is working as expected and take action if some component isn't.

6. “Those who cannot remember the past are condemned to repeat it” - George Santayana So here’s a case study…

7. A long time ago in a data centre far, far away….

9. Complete system includes humans to run it! Human Factors Engineering. http://en.wikipedia.org/wiki/Human_factors 2 x Linux Engineers 1 x Network Engineer 1 x Do Anything Guy 1 x Developer

10. No Monitoring or Metrics. Black Box. Completely blind.

11. MRTG Net Saint

12.

13. Large Development team External Consultants ITIL Process people 5 x Linux Engineers 1 x Network Engineer 2 x Database Administrators and Part of an Infrastructure team that included Virtualization specialists Storage specialists Hardware specialists

14. WTF happened? It grew… Virtualization / Cloud Cloud / Virtualization

15. Approximately 400 servers Still using Nagios and Cacti 15 minutes to add server manually. 1 hour or more to add a new check.

16. And Ganglia. And External SAAS tools: New Relic. Gomez. Omniture.

17. #monitoringsucks

18. Getting it right

19. Getting it wrong

20. What’s different about the cloud? • Servers come and go • Sometimes automatically with auto-scaling • Topologies and Architectures change rapidly • Driven from Configuration Management Systems

21. The problems with Nagios • Clunky UI. • Monolithic design. • Hard to scale. • Hard to add nodes dynamically.

22. #doingitright

23. #doingitwrong

24. Sensu… Is it the Nagios killer?

25. sensu-server sensu-client sensu-api sensu-dashboard

26. • JSON everywhere • Can re-use Nagios checks • Messaging oriented architecture • Designed to be driven from Config Management tools • Supports dynamic topologies

27. ?

28. ?

29. David Lutz 99designs @dlutzy meetup.com/Infrastructure-Coders/

Monitoring and metrics in the cloud

Recommended

Recommended

More Related Content

Similar to Monitoring and metrics in the cloud

Similar to Monitoring and metrics in the cloud (20)

More from David Lutz

More from David Lutz (6)

Recently uploaded

Recently uploaded (20)

Monitoring and metrics in the cloud