How to Monitor Microservices

•

4 likes•3,465 views

How to Monitor Microservices running in Docker, Kubernetes, Openshift, DC/OS Mesos or any other container orchestration platform.

Software

Jorge Salamero Sanz
@bencerillo @sysdig
How to
Monitor Microservices

How to Monitor Microservices?
Apps
Infra
Health
Checks
JVM/JMX
Custom
metrics
Metrics Processing Unicorns, rainbows
And cute dashboards

% whoami
Jorge Salamero Sanz
<jorge.salamero@sysdig.com>
• Working on OSS last 12 years
• Working on monitoring last 3 years
• Containers gamer @sysdig
@bencerillo
@sysdig

Agenda
• Challenges of container infrastructures
• Traditional monitoring limitations
• Best practices monitoring Microservices
• Sysdig, container native monitoring & troubleshooting

Breaking traditional model
Microservices and containers break traditional monitoring and
troubleshooting models

Traditional deployment
Full host OS
kernel
systemd
syslogd
App services
MySQL
Nginx
OpenSSL
Java
App A
App B
App C
Ops Devs

Application workflow
Database App Cache
backend middleware frontend

The Docker revolution
Servers Virtual Machines Containers
Unit: machine
Orchestration: no
Architecture:
monolithic
Unit: machine
Orchestration:
external
Architecture:
monolithic
Unit: (micro)service
Orchestration:
native
Architecture:
distributed

Containerized deployment
Full host OS
kernel
+
Docker
MySQL App A
Ops DevOps
Nginx + OpenSSL App B
Java 8.0 build XXX App C

… but in reality
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node

Container monitoring
New challenges:
1. How do we get the metrics?
2. How do we shape all this amount of metrics?
3. Analysis and troubleshooting
4. Teams on Microservices infrastructure

1. Metric collection
• We containers, because:
– are simple
– are small
– are isolated
– less dependencies
• … but they are an opaque blackbox

“Workarounds”
Agent in the
Docker container
Agent in the
Kubernetes pod
Export metrics through
an external agent
App Agent App Agent
App
Agent
App
App
App
1. Complex instrumentation (x2 because just the monitoring) plus
service monitoring configuration
2. Limited and pre-established metric collection (Docker API, etc)

Kernel instrumentation
Kernel
Docker
Container
1
Container
2
Container
3
App App
rkt LXC
Sysdig
Docker

Why this is cool?
• Just one instrumentation per host:
– spawning or destroying a container is instrumentation-less
• Full visibility: all the system calls:
– automatic service discovery
– all metrics collection (no filtering)
– application monitoring without instrumentation (magic of
decoding protocols)

Remember... but in reality:
Database App Cache/Frontend
Computing node
Computing node Computing node
Computing node Computing node
Computing node

2. Information aggregation
• Infrastructure monitoring should be transparent and
automatic (no instrumentation no configuration)
• You should handle your custom/biz metrics
• All metrics should be tagged automatically
• All metrics should be aggregated and segmented on a
service level basis

Orchestration platforms knows already...

Real example
https://github.com/kubernetes/kubernetes/issues/1405
1

3. Analysis & troubleshooting
• Imagine:
strace + wireshark + htop + lsof + iostat + vmstat + *
• Not available on containers, don’t understand
namespaces
• Metrics and logs can bite your in the ass, system
calls have all the truth
• Infrastructure gets more complex and volatile

Teams on Microservices
Francesc Zacarias, SRE @ Spotify

4. Teams by service
• Tags/Metadata from the orchestration platform, eg
Kubernetes:
– namespaces (dev, prod)
– services, deployments, RCs, pods
– custom tags
• ACLs out of the box (dashboards, alerts, etc) on
multi-tenant/PaaS scenarios

Sysdig
• 100% open-source
• 1M+ downloads
• Host analysis
• sysdig.org
• SaaS & on-prem
• 200+ customers
• Cluster analysis
• Dashboards, alerts,
events, teams
• sysdig.com

¡Danke
MicroXchg!
@bencerillo
@sysdig
www.sysdig.org
www.sysdig.com

Continuous Delivery helps to keep your software and Docker images updated and deploy new versions in production easily. Microservices are great at reducing the attack vector and limiting the privileges or credentials access of each piece of your application. Containers provide an opportunity to implement better security, small, immutable, single process and purpose. In this session, we will discover real use case examples on how to make your CI/CD pipeline interact with Docker security tools. But security doesn’t stop where your deployment pipeline ends. How can we prepare for 0-days and policy violations that happen at run-time? Can we make it part of the CI/CD process?

Sysdig monitor - a brief introduction

Daniel Kerwin

The Sysdig Secure DevOps Platform

Ashnikbiz

Proactive ops for container orchestration environments

Docker, Inc.

Break -> inspect -> fix is the Ops workflow for infrastructure stacks of the past. Distributed infrastructure and applications claim to be the new generation, but why is it so much more painful to maintain and troubleshoot them? Much of the pain comes from outdated operational models relying on reactive or, worse yet, manual monitoring and Ops. This talk lays out a proactive Ops model for container infrastructure. By focusing on event monitoring, infrastructure state monitoring, trend analysis, and distributed log collection, a proactive Ops model delivers observability for distributed apps that was not possible before. Using real-world examples from Swarm and Kubernetes, we'll demonstrate the tools used and how we relieve Ops pain in container orchestration.

15 kubernetes failure points you should watch

Sysdig

DockerCon EU 2015: The Latest in Docker Engine

Docker, Inc.

Docker on docker leveraging kubernetes in docker ee

Docker, Inc.

Two things are for certain – space is hard, and Docker is not just for web content! Space software development traditionally lags behind state of the art software process for good reason – our missions are long (7+ years), we run on highly constrained embedded hardware, and the software cannot fail. Docker, along with a devops mentality, has helped us create a scalable, parallelizable and rapidly deployable test infrastructure for DART, NASA’s mission to hit an asteroid at 6 km/s. During the presentation, we will walk through how our dev cycle has changed from a human based testing system to an automated one. We will outline how we are using Docker (and NASA Goddard’s Core Flight Executive) for both our embedded development environment and our scalable test environment. Next, we will discuss what deployment means to us (and how different it is from web deployment). Lastly, we will explore lessons learned on how our hardware-centric testing approach was adapted into a software-based approach: what worked, what didn’t, what we wish we could do someday. How can you help? We are new to Docker. We are excited to share our experiences and hear from the Docker community on our use cases, technological hurdles that we faced, our solutions to these problems, and how we can harness Docker to the fullest extent.

Configuration Management and Transforming Legacy Applications in the Enterpri...

Docker, Inc.

Share the continuity of Société Générale's journey with Docker Enterprise from different points of view, from executives to devops, with CD platform as an enabler. Creating a Dockerfile that runs a container on a developer's laptop is pretty straightforward. But extending that to stacks of containers running on a dozen environments (development, integration, testing, staging, production, etc.) with different configuration and topologies can be a challenge. This talk will cover aspects of our journey to Docker Enterprise: What configuration should go in an image? Where to put different types of configuration? Images, environment variables, entrypoint, ...? How to store assets for building images and configuration for deployment in version control. We will discuss how Société Générale has implemented these, and what we plan next for Docker Enterprise deployment.

ContainerCon sysdig Slides

Loris Degioanni

Kubernetes 架構與虛擬化之差異

inwin stack

Effective Data Pipelines with Docker & Jenkins - Brian Donaldson

Docker, Inc.

Ever find yourself needing data pipelines to feed a hungry data-driven culture, but not sure where to start, or what features are essential? In this talk, I will demonstrate a baseline data pipeline infrastructure built with Jenkins and Docker EE that checks all the boxes. Data pipelines often exist as that mysterious plumbing buried underground: occasionally inspected, but largely prone to silent failures and the ensuing hot fixes. Join the quest to daylight the infrastructure and benefit!

Chaos Engineering for Docker

Alexei Ledenev

Experiences with AWS immutable deploys and job processing

Docker, Inc.

How Docker is used at Gilt: At Gilt we use Docker primarily as a unit of immutability and to allow a standard way of deploying all kinds of software as opposed to its container properties. Why Gilt built Ionroller: An overview of the problems we tried to solve with Ionroller and immutable deploys. Pitfalls we've encountered with immutable deployments since Ionroller saw adoption in Gilt. Will cover issues such as DNS traffic migration, utilisation of resources ELBs not warmed up properly, Elasticbeanstalk using Nginx as proxy etc. Our experiences with Cloudformation and Codedeploy as an alternative to Ionroller and Elasticbeanstalk. Jobs: How we used to do batch jobs. Solutions we considered such as Mesos and Chronos. An overview of Sundial, an in house solution we built in the last few months and hope to open source for running containerized Docker jobs on Amazon ECS and why we chose it as our preferred solution.

containerd the universal container runtime

Docker, Inc.

containerd is an industry-standard core container runtime with an emphasis on simplicity, robustness and portability. It is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc.. containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users. containerd includes a daemon exposing gRPC API over a local UNIX socket. The API is a low-level one designed for higher layers to wrap and extend. It also includes a barebone CLI (ctr) designed specifically for development and debugging purpose. It uses runC to run containers according to the OCI specification. The code can be found on GitHub, and here are the contribution guidelines. containerd is based on the Docker Engine’s core container runtime to benefit from its maturity and existing contributors.

Introducing LinuxKit

Docker, Inc.

Containerd Donation to CNCF Cloud Native Conference Berlin 2017

Patrick Chanezon

DockerCon EU 2015: Cultural Revolution - How to Mange the Change Docker Brings

Docker, Inc.

Presentation by Ian Miell, author of 'Docker in Practice' The adoption of Docker brings with it many challenges, not all of themtechnical. There is a clear vision of Docker regularly articulated thatencompasses microservices, decoupled applications, agile and continuous deliveryand integration, and DevOps. But the application of this vision often flounders when confronted by the legacy structures of working and successful businesses. This talk draws on the experienced gained from several attempts to lead changewithin technical organisations. The speaker experienced both failure to effectchange (through productization of a bespoke software stack, and the introduction of Erlang), and success (moving to a Continuous Improvement modelof complex software maintenance). These experiences informed a successful drive to Docker as the build and delivery system of a 700-strong software company,with significant efficiency improvements. As with most historical revolutions, the integration of the vision with realityresulted in corners cut and principles compromised in order to effect change.This talk is a discussion of the lessons learned from these experiences in an effort to help the listener clarify and overcome the unique challenges their own organisation brings to making change happen.

Deploying Kubernetes without scaring off your security team - KubeCon 2017

Major Hayden

From Code to Kubernetes

Daniel Oliveira Filho

Slides from the talk given to the Startup Berlin Slack Group that demonstrates how TruckIN is implementing its continuous delivery workflow using technologies and open-source tools. Topics that are covered: Automated Cloud Provisioning (Network, Subnets, VMs, Kubernetes Cluster, Firewall, Disks, Credentials, Private Docker Registry); Configuration Management (Salt Stack), Continuous Integration (Jenkins CI), Continuous Delivery/Deployment (Salt API/Reactor + Kubernetes) to a Google Cloud Kubernetes Cluster, Remote Application Debugging, Managing Google Cloud Kubernetes Cluster, Logging, Monitoring and ChatOps (Slack and operable.io)

DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP

Thomas Graf

This talk will start with a deep dive and hands on examples of BPF, possibly the most promising low level technology to address challenges in application and network security, tracing, and visibility. We will discuss how BPF evolved from a simple bytecode language to filter raw sockets for tcpdump to the a JITable virtual machine capable of universally extending and instrumenting both the Linux kernel and user space applications. The introduction is followed by a concrete example of how the Cilium open source project applies BPF to solve networking, security, and load balancing for highly distributed applications. We will discuss and demonstrate how Cilium with the help of BPF can be combined with distributed system orchestration such as Docker to simplify security, operations, and troubleshooting of distributed applications.

Troubleshooting Tips from a Docker Support Engineer

Jeff Anderson

What Have Namespaces Done for you Lately? Liz Rice, Aqua Security

Docker, Inc.

Kubernetes security

Saiyam Pathak

Docker for Ops - Scott Coulton, Puppet

Docker, Inc.

In this talk, Scott Coulton will take you through Docker's cluster solution Swarm mode with his operations hat on. We will start from the beginning by describing what swarm mode is, what it does, and how it works behind the scenes. From there, we will look at very basic configurations of Swarm mode from the point of view of the operations team as well as a production-ready workflow including deployments of the cluster, logging and CD best practices. Attendees will be able to apply their learnings to their use cases.

Continuous Delivery With Containers

All Things Open

Building Containers: How Many Ways Are Too Many?

Vietnam Open Infrastructure User Group

Monitoring microservices: Docker, Mesos and Kubernetes visibility at scale

Alessandro Gallotta

Microservices and containers are revolutionizing the way we deploy applications and maintain infrastructure. But as many have found containers still have a key problem: monitoring and troubleshooting them can be impractical, painful, and sometimes impossible. With the rise of microservice based architectures and orchestration tools such as Kubernetes and Mesos, managing this has become even harder. Using real tools, in live environments, Alessandro Gallotta will walk through various hands-on scenarios including how to: -visualize physical vs logical architectures of Kubernetes/Mesos deployments -understand performance at the microservice/app level for orchestrated systems -identify & surface system activity of individual Docker containers -extract process & app-level metrics inside containers with non-intrusive methods -troubleshoot detailed network activity in distributed containers

ThroughTheLookingGlass_EffectiveObservability.pptx

Grace Jansen

Our cloud-native environments are more complex than ever before! So how can we ensure that the applications we’re deploying to them are behaving as we intended them to? This is where effective observability is crucial. It enables us to monitor our applications in real-time and analyse and diagnose their behaviour in the cloud. However, until recently, we were lacking the standardization to ensure our observability solutions were applicable across different platforms and technologies. In this session, we’ll delve into what effective observability really means, exploring open source technologies and specifications, like OpenTelemetry, that can help us to achieve this while ensuring our applications remain flexible and portable.

What's hot

Automated hardware testing using docker for space

Docker, Inc.

Configuration Management and Transforming Legacy Applications in the Enterpri...

Docker, Inc.

ContainerCon sysdig Slides

Loris Degioanni

Kubernetes 架構與虛擬化之差異

inwin stack

Effective Data Pipelines with Docker & Jenkins - Brian Donaldson

Docker, Inc.

Chaos Engineering for Docker

Alexei Ledenev

Experiences with AWS immutable deploys and job processing

Docker, Inc.

containerd the universal container runtime

Docker, Inc.

Introducing LinuxKit

Docker, Inc.

Containerd Donation to CNCF Cloud Native Conference Berlin 2017

Patrick Chanezon

DockerCon EU 2015: Cultural Revolution - How to Mange the Change Docker Brings

Docker, Inc.

Deploying Kubernetes without scaring off your security team - KubeCon 2017

Major Hayden

From Code to Kubernetes

Daniel Oliveira Filho

DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP

Thomas Graf

Troubleshooting Tips from a Docker Support Engineer

Jeff Anderson

What Have Namespaces Done for you Lately? Liz Rice, Aqua Security

Docker, Inc.

Kubernetes security

Saiyam Pathak

Docker for Ops - Scott Coulton, Puppet

Docker, Inc.

Continuous Delivery With Containers

All Things Open

Building Containers: How Many Ways Are Too Many?

Vietnam Open Infrastructure User Group

What's hot (20)

Automated hardware testing using docker for space

Configuration Management and Transforming Legacy Applications in the Enterpri...

ContainerCon sysdig Slides

Kubernetes 架構與虛擬化之差異

Effective Data Pipelines with Docker & Jenkins - Brian Donaldson

Chaos Engineering for Docker

Experiences with AWS immutable deploys and job processing

containerd the universal container runtime

Introducing LinuxKit

Containerd Donation to CNCF Cloud Native Conference Berlin 2017

DockerCon EU 2015: Cultural Revolution - How to Mange the Change Docker Brings

Deploying Kubernetes without scaring off your security team - KubeCon 2017

From Code to Kubernetes

DockerCon 2017 - Cilium - Network and Application Security with BPF and XDP

Troubleshooting Tips from a Docker Support Engineer

What Have Namespaces Done for you Lately? Liz Rice, Aqua Security

Kubernetes security

Docker for Ops - Scott Coulton, Puppet

Continuous Delivery With Containers

Building Containers: How Many Ways Are Too Many?

Similar to How to Monitor Microservices

Monitoring microservices: Docker, Mesos and Kubernetes visibility at scale

Alessandro Gallotta

ThroughTheLookingGlass_EffectiveObservability.pptx

Grace Jansen

DockerCon EU 2015: Monitoring Docker

Docker, Inc.

Operational Visibiliy and Analytics - BU Seminar

Canturk Isci

Big Brother: Kubernetes Edition

Knox Anderson

In this short demo-driven meetup, we'll help you get a handle on what's changing and how it will impact your DevOps practice. We'll cover: - What are the operational limitations of containers in production? - How do you get visibility inside containers without super-human effort? - How do you look into kubernetes performance, and not just container performance? - A live install of Sysdig Cloud on a running environment

Weave User Group Talk - DockerCon 2017 Recap

Patrick Chanezon

Docker moves very fast, with an edge channel released every month and a stable release every 3 months. Patrick will talk about how Docker introduced Docker EE and a certification program for containers and plugins with Docker CE and EE 17.03 (from March), the announcements from DockerCon (April), and the many new features planned for Docker CE 17.05 in May. This talk will be about what's new in Docker and what's next on the roadmap

The Art of Container Monitoring

Derek Chen

According to service scale, there are hundreds or thousands of running containers in your service. Should we monitor each container by microscope or monitor each microservice by magnifier? This depends which granularity can help us find and solve the problems. In this sharing, I will introduce how to use cAdvisor, Icinga2, InfluxDB and Grafana to build a self-hosted monitoring system. In addition, I also discuss with how to embrace open source and share some practical experiences.

Kaseya Connect 2012 - THE ABC'S OF MONITORING

Kaseya

Integration in the Cloud, by Rob Davies

Judy Breedlove

Kubernetes Monitoring & Best Practices

Ajeet Singh Raina

DCSF19 Container Security: Theory & Practice at Netflix

Docker, Inc.

Michael Wardrop, Netflix Usage of containers has undergone rapid growth at Netflix and it is still accelerating. Our container story started organically with developers downloading Docker and using it to improve their developer experience. The first production workloads were simple batch jobs, pioneering micro-services followed, then status as a first class platform running critical workloads. As the types of workloads changed and their importance increased, the security of our container ecosystem needed to evolve and adapt. This session will cover some security theory, architecture, along with practical considerations, and lessons we learnt along the way.

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015

Datadog

Monitoring Containerized Micro-Services In Azure

Alex Bulankou

Microservices Architecture - Cloud Native Apps

Araf Karsh Hamid

Containers as Infrastructure for New Gen Apps

Khalid Ahmed

Breaking the Monolith Road to Containers

Amazon Web Services

Following simple patterns of good application design can allow you to scale your application for your customers easily. This presentation dives into the 12 factor application design and demo how this applies to containers and deployments on Amazon ECS and Fargate. We'll take a look at tooling that can be used to simplify your workflow and help you adopt the principles of the 12 factor application.

Monitoring in Motion: Monitoring Containers and Amazon ECS

Amazon Web Services

Patterns and Pains of Migrating Legacy Applications to Kubernetes

QAware GmbH

Open Source Summit 2018, Vancouver (Canada): Talk by Josef Adersberger (@adersberger, CTO at QAware), Michael Frank (Software Architect at QAware) and Robert Bichler (IT Project Manager at Allianz Germany) Abstract: Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud-native apps. But what to do if you’ve no shiny new cloud-native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can! We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year. The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.

Patterns and Pains of Migrating Legacy Applications to Kubernetes

Josef Adersberger

Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can! We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year. The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.

Kubernetes @ meetic

Sébastien Le Gall

3 years ago, Meetic chose to rebuild it's backend architecture using microservices and an event driven strategy. As we where moving along our old legacy application, testing features became gradually a pain, especially when those features rely on multiple changes across multiple components. Whatever the number of application you manage, unit testing is easy, as well as functional testing on a microservice. A good gherkin framework and a set of docker container can do the job. The real challenge is set in end-to-end testing even more when a feature can involve up to 60 different components. To solve that issue, Meetic is building a Kubernetes strategy around testing. To do such a thing we need to : - Be able to generate a docker container for each pull-request on any component of the stack - Be able to create a full testing environment in the simplest way - Be able to launch automated test on this newly created environment - Have a clean-up process to destroy testing environment after tests To separate the various testing environment, we chose to use Kubernetes Namespaces each containing a variant of the Meetic stack. But when it comes to Kubernetes, managing multiple namespaces can be hard. Yaml configuration files need to be shared in a way that each people / automated job can access to them and modify them without impacting others. This is typically why Meetic chose to develop it's own tool to manage namespace through a cli tool, or a REST API on which we can plug a friendly UI. In this talk we will tell you the story of our CI/CD evolution to satisfy the need to create a docker container for each new pull request. And we will show you how to make end-to-end testing easier using Blackbeard, the tool we developed to handle the need to manage namespaces inspired by Helm.

Similar to How to Monitor Microservices (20)

Monitoring microservices: Docker, Mesos and Kubernetes visibility at scale

ThroughTheLookingGlass_EffectiveObservability.pptx

DockerCon EU 2015: Monitoring Docker

Operational Visibiliy and Analytics - BU Seminar

Big Brother: Kubernetes Edition

Weave User Group Talk - DockerCon 2017 Recap

The Art of Container Monitoring

Kaseya Connect 2012 - THE ABC'S OF MONITORING

Integration in the Cloud, by Rob Davies

Kubernetes Monitoring & Best Practices

DCSF19 Container Security: Theory & Practice at Netflix

Docker Usage Patterns - Meetup Docker Paris - November, 10th 2015

Monitoring Containerized Micro-Services In Azure

Microservices Architecture - Cloud Native Apps

Containers as Infrastructure for New Gen Apps

Breaking the Monolith Road to Containers

Monitoring in Motion: Monitoring Containers and Amazon ECS

Patterns and Pains of Migrating Legacy Applications to Kubernetes

Kubernetes @ meetic

More from Sysdig

Wordpress y Docker, de desarrollo a produccion

Sysdig

Docker esta revolucionando cómo desplegamos nuestras aplicaciones. Desde el entorno de desarrollo hasta la puesta en producción. Veremos las ventajas que nos aporta Docker para el desarrollo en WordPress, las herramientas y procesos desde el punto de vista de un desarrollador. A la hora de mover nuestra aplicación WordPress a producción, presentaremos los retos que presenta y las ventajas que aportan herramientas de orquestación como Kubernetes. Tanto si eres un desarrollador como si también tienes que gestionar los sistemas que alojan tu WordPress, saldrás de esta charla queriendo poner todos tus WordPress en contenedores.

What Prometheus means for monitoring vendors

Sysdig

Docker Runtime Security

Sysdig

While there have been many improvements around securing containers, there is still a large gap in monitoring the behaviour of containers in production. Sysdig Falco is an open source behavioural activity monitor for containerized environments. Sysdig Falco can detect and alert on anomalous behaviour at the application, file, system, and network level. In this session get a deep dive into Falco: How does behavioural security differ from existing security solutions like image scanning, seccomp, SELinux or AppArmor? What can Sysdig Falco detect? Building and customizing rules for your Docker and Kubernetes apps. Forensics analysis with Sysdig Inspect even when the container doesn't exist anymore! Read more on: https://sysdig.com/blog/docker-runtime-security/ https://sysdig.com/blog/runtime-security-kubernetes-sysdig-falco/

Continuous Security

Sysdig

Continuous Delivery helps to keep your software and Docker images updated and deploy new versions in production easily. Microservices are great reducing the attack vector and limiting the privileges or credentials access of each piece of your application. Containers provide an opportunity to implement better security, small, immutable, single process and purpose. In this session, we will discover real use case examples on how to make your CI/CD pipeline interact with Docker security tools. But security doesn’t stop where your deployment pipeline ends. How we can prepare for 0-days and policy violations that happen at run-time? Can we make it part of the CI/CD process?

The top 5 Kubernetes metrics to monitor

Sysdig

The top 5 Kubernetes metrics to monitor

Sysdig

Behavioural activity monitoring on CoreOS with Sysdig Falco

Sysdig

WTF my container just spawned a shell!

Sysdig

Trace everything, when APM meets SysAdmins

Sysdig

The Dark Art of Container Monitoring - Spanish

Sysdig

Lions, Tigers and Deers: What building zoos can teach us about securing micro...

Sysdig

Building Trustworthy Containers

Sysdig

Containers have the potential to improve the security of typical deployments, but for many the argument has not yet been made convincingly. This talk will describe the existing security technologies around containers, and show how their use can make container-based systems more secure than the alternatives. It will then go further, describing new technologies that allow admins to have even greater confidence in the security of their systems, beyond anything possible with traditional deployment techniques.

A brief history of system calls

Sysdig

System calls are the primary mechanism of user-to-kernel interaction. Today the Linux system call interface has achieved a primacy and ubiquity that make it an ideal layer at which to understand single-system and distributed-system pathologies. Sysdig advances the art of system call observability by drawing on the systems that came before it. Informed by his work with /proc, process tools and DTrace, Adam will walk through a history of system calls and system call observability from simple systems like truss and strace, moderns ones like DTrace and SystemTab, and ancient ones from the early days of Unix.

Designing Tracing Tools

Sysdig

You have a system with an advanced programmatic tracer: do you know what to do with it? Brendan has used numerous tracers in production environments, and has published hundreds of tracing-based tools. In this talk he will share tips and know-how for creating CLI tracing tools and GUI visualizations, to solve real problems effectively. Programmatic tracing is an amazing superpower, and this talk will show you how to wield it!

Extending Sysdig with Chisel

Sysdig

Intro to sysdig in 15 minutes

Sysdig

Troubleshooting Kubernetes

Sysdig

Kubernetes is a tremendous system for orchestrating your containers onto physical infrastructure. But troubleshooting Kubernetes can be incredibly challenging due to the dynamic and isolated nature of the containers it orchestrates. Sysdig leverages the powerful concept of container-aware system events and correlates each one of them with super rich metadata Kubernetes. In this session you’ll go deep into a couple of Kubernetes issues and how you would track them down using sysdig.

Find the Hacker

Sysdig

How to Secure Containers

Sysdig

While there have been many improvements around improving containers, there is still a large gap in securing the behavior of containers in production. Enter sysdig falco, the behavioral activity monitor for containerized environments. It can detect and alert on anomalous behavior at the application, file, system, and network level. In this session get a deep dive into falco and learn: - How does behavioral security differ from existing security solutions like image scanning? - How does falco work? What can it detect? - How do you build and customize rules for falco?

Sysdig Meetup - San Francisco, December 2014

Sysdig

More from Sysdig (20)

Wordpress y Docker, de desarrollo a produccion

What Prometheus means for monitoring vendors

Docker Runtime Security

Continuous Security

The top 5 Kubernetes metrics to monitor

Behavioural activity monitoring on CoreOS with Sysdig Falco

WTF my container just spawned a shell!

Trace everything, when APM meets SysAdmins

The Dark Art of Container Monitoring - Spanish

Lions, Tigers and Deers: What building zoos can teach us about securing micro...

Building Trustworthy Containers

A brief history of system calls

Designing Tracing Tools

Extending Sysdig with Chisel

Intro to sysdig in 15 minutes

Troubleshooting Kubernetes

Find the Hacker

How to Secure Containers

Sysdig Meetup - San Francisco, December 2014

Recently uploaded

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

WSO2

Providing Globus Services to Users of JASMIN for Environmental Data Analysis

Globus

JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...

Globus

The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.

Globus Compute Introduction - GlobusWorld 2024

Globus

Enhancing Research Orchestration Capabilities at ORNL.pdf

Globus

Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.

Cracking the code review at SpringIO 2024

Paco van Beckhoven

Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production. Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process? In this session we will cover: - The Art of Effective Code Reviews - Streamlining the Review Process - Elevating Reviews with Automated Tools By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces

GlobusWorld 2024 Opening Keynote session

Globus

Explore Modern SharePoint Templates for 2024

Sharepoint Designs

Using IESVE for Room Loads Analysis - Australia & New Zealand

IES VE

Visitor Management System in India- Vizman.app

NaapbooksPrivateLimi

Your Digital Assistant. Making complex approach simple. Straightforward process saves time. No more waiting to connect with people that matter to you. Safety first is not a cliché - Securely protect information in cloud storage to prevent any third party from accessing data. Would you rather make your visitors feel burdened by making them wait? Or choose VizMan for a stress-free experience? VizMan is an automated visitor management system that works for any industries not limited to factories, societies, government institutes, and warehouses. A new age contactless way of logging information of visitors, employees, packages, and vehicles. VizMan is a digital logbook so it deters unnecessary use of paper or space since there is no requirement of bundles of registers that is left to collect dust in a corner of a room. Visitor’s essential details, helps in scheduling meetings for visitors and employees, and assists in supervising the attendance of the employees. With VizMan, visitors don’t need to wait for hours in long queues. VizMan handles visitors with the value they deserve because we know time is important to you. Feasible Features One Subscription, Four Modules – Admin, Employee, Receptionist, and Gatekeeper ensures confidentiality and prevents data from being manipulated User Friendly – can be easily used on Android, iOS, and Web Interface Multiple Accessibility – Log in through any device from any place at any time One app for all industries – a Visitor Management System that works for any organisation. Stress-free Sign-up Visitor is registered and checked-in by the Receptionist Host gets a notification, where they opt to Approve the meeting Host notifies the Receptionist of the end of the meeting Visitor is checked-out by the Receptionist Host enters notes and remarks of the meeting Customizable Components Scheduling Meetings – Host can invite visitors for meetings and also approve, reject and reschedule meetings Single/Bulk invites – Invitations can be sent individually to a visitor or collectively to many visitors VIP Visitors – Additional security of data for VIP visitors to avoid misuse of information Courier Management – Keeps a check on deliveries like commodities being delivered in and out of establishments Alerts & Notifications – Get notified on SMS, email, and application Parking Management – Manage availability of parking space Individual log-in – Every user has their own log-in id Visitor/Meeting Analytics – Evaluate notes and remarks of the meeting stored in the system Visitor Management System is a secure and user friendly database manager that records, filters, tracks the visitors to your organization. "Secure Your Premises with VizMan (VMS) – Get It Now"

Accelerate Enterprise Software Engineering with Platformless

WSO2

Key takeaways: Challenges of building platforms and the benefits of platformless. Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience. How Choreo enables the platformless experience. How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo. Demo of an end-to-end app built and deployed on Choreo.

Prosigns: Transforming Business with Tailored Technology Solutions

Prosigns

Unlocking Business Potential: Tailored Technology Solutions by Prosigns Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support. Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth. Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices. AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making. Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency. DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration. Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly. Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business. Join us on a journey of innovation and growth. Let's partner for success with Prosigns.

BoxLang: Review our Visionary Licenses of 2024

Ortus Solutions, Corp

Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf

AMB-Review

Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos https://www.amb-review.com/tubetrivia-ai Exclusive Features: AI-Powered Questions, Wide Range of Categories, Adaptive Difficulty, User-Friendly Interface, Multiplayer Mode, Regular Updates. #TubeTriviaAI #QuizVideoMagic #ViralQuizVideos #AIQuizGenerator #EngageExciteExplode #MarketingRevolution #BoostYourTraffic #SocialMediaSuccess #AIContentCreation #UnlimitedTraffic

Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL

Natan Silnitsky

In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey. Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience. Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system. Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.

Designing for Privacy in Amazon Web Services

KrzysztofKkol1

Data privacy is one of the most critical issues that businesses face. This presentation shares insights on the principles and best practices for ensuring the resilience and security of your workload. Drawing on a real-life project from the HR industry, the various challenges will be demonstrated: data protection, self-healing, business continuity, security, and transparency of data processing. This systematized approach allowed to create a secure AWS cloud infrastructure that not only met strict compliance rules but also exceeded the client's expectations.

Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...

Anthony Dahanne

Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ? Venez le découvrir lors de cette session ignite

Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...

Globus

Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.

Globus Compute wth IRI Workflows - GlobusWorld 2024

Globus

As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.

De mooiste recreatieve routes ontdekken met RouteYou en FME

Jelle | Nordend

Recently uploaded (20)

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

Providing Globus Services to Users of JASMIN for Environmental Data Analysis

Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...

Globus Compute Introduction - GlobusWorld 2024

Enhancing Research Orchestration Capabilities at ORNL.pdf

Cracking the code review at SpringIO 2024

GlobusWorld 2024 Opening Keynote session

Explore Modern SharePoint Templates for 2024

Using IESVE for Room Loads Analysis - Australia & New Zealand

Visitor Management System in India- Vizman.app

Accelerate Enterprise Software Engineering with Platformless

Prosigns: Transforming Business with Tailored Technology Solutions

BoxLang: Review our Visionary Licenses of 2024

Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf

Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL

Designing for Privacy in Amazon Web Services

Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...

Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...

Globus Compute wth IRI Workflows - GlobusWorld 2024

De mooiste recreatieve routes ontdekken met RouteYou en FME

How to Monitor Microservices

1. Jorge Salamero Sanz @bencerillo @sysdig How to Monitor Microservices

2. How to Monitor Microservices? Apps Infra Health Checks JVM/JMX Custom metrics Metrics Processing Unicorns, rainbows And cute dashboards

4. % whoami Jorge Salamero Sanz <jorge.salamero@sysdig.com> • Working on OSS last 12 years • Working on monitoring last 3 years • Containers gamer @sysdig @bencerillo @sysdig

5. Agenda • Challenges of container infrastructures • Traditional monitoring limitations • Best practices monitoring Microservices • Sysdig, container native monitoring & troubleshooting

6. Breaking traditional model Microservices and containers break traditional monitoring and troubleshooting models

7. Traditional deployment Full host OS kernel systemd syslogd App services MySQL Nginx OpenSSL Java App A App B App C Ops Devs

8. Application workflow Database App Cache backend middleware frontend

9. The Docker revolution Servers Virtual Machines Containers Unit: machine Orchestration: no Architecture: monolithic Unit: machine Orchestration: external Architecture: monolithic Unit: (micro)service Orchestration: native Architecture: distributed

10. Containerized deployment Full host OS kernel + Docker MySQL App A Ops DevOps Nginx + OpenSSL App B Java 8.0 build XXX App C

11. … but in reality Database App Cache/Frontend Computing node Computing node Computing node Computing node Computing node Computing node

12. … but in reality Database App Cache/Frontend Computing node Computing node Computing node Computing node Computing node Computing node

13. Application workflow Database App Cache backend middleware frontend

14. New organization structures

15. Container monitoring New challenges: 1. How do we get the metrics? 2. How do we shape all this amount of metrics? 3. Analysis and troubleshooting 4. Teams on Microservices infrastructure

16. Container monitoring New challenges: 1. How do we get the metrics? 2. How do we shape all this amount of metrics? 3. Analysis and troubleshooting 4. Teams on Microservices infrastructure

17. 1. Metric collection • We containers, because: – are simple – are small – are isolated – less dependencies • … but they are an opaque blackbox

18. “Workarounds” Agent in the Docker container Agent in the Kubernetes pod Export metrics through an external agent App Agent App Agent App Agent App App App 1. Complex instrumentation (x2 because just the monitoring) plus service monitoring configuration 2. Limited and pre-established metric collection (Docker API, etc)

19. Kernel instrumentation Kernel Docker Container 1 Container 2 Container 3 App App rkt LXC Sysdig Docker

20. Why this is cool? • Just one instrumentation per host: – spawning or destroying a container is instrumentation-less • Full visibility: all the system calls: – automatic service discovery – all metrics collection (no filtering) – application monitoring without instrumentation (magic of decoding protocols)

21. Container monitoring New challenges: 1. How do we get the metrics? 2. How do we shape all this amount of metrics? 3. Analysis and troubleshooting 4. Teams on Microservices infrastructure

22. Remember... but in reality: Database App Cache/Frontend Computing node Computing node Computing node Computing node Computing node Computing node

23. 2. Information aggregation • Infrastructure monitoring should be transparent and automatic (no instrumentation no configuration) • You should handle your custom/biz metrics • All metrics should be tagged automatically • All metrics should be aggregated and segmented on a service level basis

24. Application workflow Database App Cache backend middleware frontend

25. Orchestration platforms knows already...

26. Demo!

27. Container monitoring New challenges: 1. How do we get the metrics? 2. How do we shape all this amount of metrics? 3. Analysis and troubleshooting 4. Teams on Microservices infrastructure

28. Real example https://github.com/kubernetes/kubernetes/issues/1405 1

29. 3. Analysis & troubleshooting • Imagine: strace + wireshark + htop + lsof + iostat + vmstat + * • Not available on containers, don’t understand namespaces • Metrics and logs can bite your in the ass, system calls have all the truth • Infrastructure gets more complex and volatile

30. Demo!

31. Container monitoring New challenges: 1. How do we get the metrics? 2. How do we shape all this amount of metrics? 3. Analysis and troubleshooting 4. Teams on Microservices infrastructure

32. Teams on Microservices Francesc Zacarias, SRE @ Spotify

33. 4. Teams by service • Tags/Metadata from the orchestration platform, eg Kubernetes: – namespaces (dev, prod) – services, deployments, RCs, pods – custom tags • ACLs out of the box (dashboards, alerts, etc) on multi-tenant/PaaS scenarios

34. Container monitoring New challenges: 1. How do we get the metrics? 2. How do we shape all this amount of metrics? 3. Analysis and troubleshooting 4. Teams on Microservices infrastructure

35. Sysdig • 100% open-source • 1M+ downloads • Host analysis • sysdig.org • SaaS & on-prem • 200+ customers • Cluster analysis • Dashboards, alerts, events, teams • sysdig.com

36. ¡Danke MicroXchg! @bencerillo @sysdig www.sysdig.org www.sysdig.com

How to Monitor Microservices

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to Monitor Microservices

Similar to How to Monitor Microservices (20)

More from Sysdig

More from Sysdig (20)

Recently uploaded

Recently uploaded (20)

How to Monitor Microservices