This document summarizes a presentation about monitoring Elasticsearch in OpenShift. It discusses the challenges of running Elasticsearch in OpenShift due to limited resources, and how Prometheus and Grafana are used to monitor Elasticsearch metrics. It also describes the Elasticsearch Operator which manages Elasticsearch clusters through custom resource definitions. Skilled personnel are needed to maintain the cluster through upgrades and troubleshooting.
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Kubernetes Webinar - Using ConfigMaps & Secrets Janakiram MSV
Many applications require configuration using some combination of configuration files, command line arguments, and environment variables. ConfigMaps in Kubernetes provide mechanisms to inject containers with configuration data while keeping them portable. Secrets decouple sensitive content from the pods using a volume plug-in. This webinar will discuss the use cases and scenarios for using ConfigMaps and Secrets.
Kubernetes Application Deployment with Helm - A beginner Guide!Krishna-Kumar
Google DevFest2019 Presentation at Infosys Campus Bangalore. Application deployment in Kubernetes with Helm is demo'ed in Google Kubernetes Engine (GKE). This is an introductory session on Helm. Several references are given in it to further explore helm3 as it is in Beta state now.
Kubernetes is designed to be an extensible system. But what is the vision for Kubernetes Extensibility? Do you know the difference between webhooks and cloud providers, or between CRI, CSI, and CNI? In this talk we will explore what extension points exist, how they have evolved, and how to use them to make the system do new and interesting things. We’ll give our vision for how they will probably evolve in the future, and talk about the sorts of things we expect the broader Kubernetes ecosystem to build with them.
OpenShift is Red Hat's Platform-as-a-Service (PaaS) that lets developers quickly develop, host, and scale Docker container-based applications. OpenShift enables a uniform and standardised approach to container management across all hosting options including AWS/EC2 and other private/public cloud and on/off-premise variants. At this session, you will learn how Red Hat's enterprise clients are using OpenShift to enable their digital transformation initiatives. Examples will cover how realising a hybrid cloud strategy can simplify and reduce the risk of migrating and transitioning application workloads to containers in the cloud.
Alex Smith, Solutions Architect, Amazon Web Services, ASEAN
Stephen Bylo, Senior Solution Architect, Red Hat Asia Pacific Pte Ltd
These are the slides for a talk/workshop delivered to the Cloud Native Wales user group (@CloudNativeWal) on 2019-01-10.
In these slides, we go over some principles of gitops and a hands on session to apply these to manage a microservice.
You can find out more about GitOps online https://www.weave.works/technologies/gitops/
A comprehensive walkthrough of how to manage infrastructure-as-code using Terraform. This presentation includes an introduction to Terraform, a discussion of how to manage Terraform state, how to use Terraform modules, an overview of best practices (e.g. isolation, versioning, loops, if-statements), and a list of gotchas to look out for.
For a written and more in-depth version of this presentation, check out the "Comprehensive Guide to Terraform" blog post series: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca
Prometheus: Monitoring by "Pravin Magdum" from "Crevise". The presentation was done at #doppa17 DevOps++ Global Summit 2017. All the copyrights are reserved with the author
Are you interested in dipping your toes in the cloud native observability waters, but as an engineer you are not sure where to get started with tracing problems through your microservices and application landscapes on Kubernetes? Then this is the session for you, where we take you on your first steps in an active open-source project that offers a buffet of languages, challenges, and opportunities for getting started with telemetry data.
The project is called openTelemetry, but before diving into the specifics, we’ll start with de-mystifying key concepts and terms such as observability, telemetry, instrumentation, cardinality, percentile to lay a foundation. After understanding the nuts and bolts of observability and distributed traces, we’ll explore the openTelemetry community; its Special Interest Groups (SIGs), repositories, and how to become not only an end-user, but possibly a contributor.We will wrap up with an overview of the components in this project, such as the Collector, the OpenTelemetry protocol (OTLP), its APIs, and its SDKs.
Attendees will leave with an understanding of key observability concepts, become grounded in distributed tracing terminology, be aware of the components of openTelemetry, and know how to take their first steps to an open-source contribution!
Key Takeaways: Open source, vendor neutral instrumentation is an exciting new reality as the industry standardizes on openTelemetry for observability. OpenTelemetry is on a mission to enable effective observability by making high-quality, portable telemetry ubiquitous. The world of observability and monitoring today has a steep learning curve and in order to achieve ubiquity, the project would benefit from growing our contributor community.
Helm - Application deployment management for KubernetesAlexei Ledenev
Use Helm to package and deploy a composed application to any Kubernetes cluster. Manage your releases easily over time and across multiple K8s clusters.
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Kubernetes Webinar - Using ConfigMaps & Secrets Janakiram MSV
Many applications require configuration using some combination of configuration files, command line arguments, and environment variables. ConfigMaps in Kubernetes provide mechanisms to inject containers with configuration data while keeping them portable. Secrets decouple sensitive content from the pods using a volume plug-in. This webinar will discuss the use cases and scenarios for using ConfigMaps and Secrets.
Kubernetes Application Deployment with Helm - A beginner Guide!Krishna-Kumar
Google DevFest2019 Presentation at Infosys Campus Bangalore. Application deployment in Kubernetes with Helm is demo'ed in Google Kubernetes Engine (GKE). This is an introductory session on Helm. Several references are given in it to further explore helm3 as it is in Beta state now.
Kubernetes is designed to be an extensible system. But what is the vision for Kubernetes Extensibility? Do you know the difference between webhooks and cloud providers, or between CRI, CSI, and CNI? In this talk we will explore what extension points exist, how they have evolved, and how to use them to make the system do new and interesting things. We’ll give our vision for how they will probably evolve in the future, and talk about the sorts of things we expect the broader Kubernetes ecosystem to build with them.
OpenShift is Red Hat's Platform-as-a-Service (PaaS) that lets developers quickly develop, host, and scale Docker container-based applications. OpenShift enables a uniform and standardised approach to container management across all hosting options including AWS/EC2 and other private/public cloud and on/off-premise variants. At this session, you will learn how Red Hat's enterprise clients are using OpenShift to enable their digital transformation initiatives. Examples will cover how realising a hybrid cloud strategy can simplify and reduce the risk of migrating and transitioning application workloads to containers in the cloud.
Alex Smith, Solutions Architect, Amazon Web Services, ASEAN
Stephen Bylo, Senior Solution Architect, Red Hat Asia Pacific Pte Ltd
These are the slides for a talk/workshop delivered to the Cloud Native Wales user group (@CloudNativeWal) on 2019-01-10.
In these slides, we go over some principles of gitops and a hands on session to apply these to manage a microservice.
You can find out more about GitOps online https://www.weave.works/technologies/gitops/
A comprehensive walkthrough of how to manage infrastructure-as-code using Terraform. This presentation includes an introduction to Terraform, a discussion of how to manage Terraform state, how to use Terraform modules, an overview of best practices (e.g. isolation, versioning, loops, if-statements), and a list of gotchas to look out for.
For a written and more in-depth version of this presentation, check out the "Comprehensive Guide to Terraform" blog post series: https://blog.gruntwork.io/a-comprehensive-guide-to-terraform-b3d32832baca
Prometheus: Monitoring by "Pravin Magdum" from "Crevise". The presentation was done at #doppa17 DevOps++ Global Summit 2017. All the copyrights are reserved with the author
Are you interested in dipping your toes in the cloud native observability waters, but as an engineer you are not sure where to get started with tracing problems through your microservices and application landscapes on Kubernetes? Then this is the session for you, where we take you on your first steps in an active open-source project that offers a buffet of languages, challenges, and opportunities for getting started with telemetry data.
The project is called openTelemetry, but before diving into the specifics, we’ll start with de-mystifying key concepts and terms such as observability, telemetry, instrumentation, cardinality, percentile to lay a foundation. After understanding the nuts and bolts of observability and distributed traces, we’ll explore the openTelemetry community; its Special Interest Groups (SIGs), repositories, and how to become not only an end-user, but possibly a contributor.We will wrap up with an overview of the components in this project, such as the Collector, the OpenTelemetry protocol (OTLP), its APIs, and its SDKs.
Attendees will leave with an understanding of key observability concepts, become grounded in distributed tracing terminology, be aware of the components of openTelemetry, and know how to take their first steps to an open-source contribution!
Key Takeaways: Open source, vendor neutral instrumentation is an exciting new reality as the industry standardizes on openTelemetry for observability. OpenTelemetry is on a mission to enable effective observability by making high-quality, portable telemetry ubiquitous. The world of observability and monitoring today has a steep learning curve and in order to achieve ubiquity, the project would benefit from growing our contributor community.
Helm - Application deployment management for KubernetesAlexei Ledenev
Use Helm to package and deploy a composed application to any Kubernetes cluster. Manage your releases easily over time and across multiple K8s clusters.
Kubernetes Basis: Pods, Deployments, and ServicesJian-Kai Wang
Kubernetes is a container management platform and empowers the scalability to the container. In this repository, we address the issues of how to use Kubernetes with real cases. We start from the basic objects in Kubernetes, Pods, deployments, and Services. This repository is also a tutorial for those with advanced containerization skills trying to step into the Kubernetes. We also provide several YAML examples for those looking for quickly deploying services. Please enjoy it and let's start the journey to Kubernetes.
Introduction to Kubernetes - Docker Global Mentor Week 2016Opsta
Kubernetes is an open-source system for automating
deployment, scaling, and management of containerized
applications. This presentation will show you overview of Kubernetes concept.
Docker Global Mentor Week 2016 #DockerInThai at Kaidee on November 18, 2016
GL DevOps Experts are committed to sharing with our community as much knowledge about Docker and Kubernetes as possible.
Thinking about Kubernetes?
Join Vadym Fabiianskiy and Andrii Mandubyra, GlobalLogic Lviv DevOps Experts and learn:
Container Runtime specifics
What are the building blocks of K8S?
How does Kubernetes work?
Deployment and release strategies
Building a modern Software as a Service platform brings a lot of interesting engineering challenges. During this talk, I’m going to share my team’s journey of building a SaaS from scratch in 2020. First, we are going to start with the technologies and the architecture we picked. Then, we’ll go over the interesting challenge of implementing multitenancy. And we'll see how we benchmarked three different options and picked one. And last but not least, we’ll explore how every startup can use open source technologies to build observability infrastructure. And how to run their SaaS in production.
In occasione del Containers & Cloud-Native Roadshow 2019 il Cloud Engineer Domenico Pastore ha fornito una panoramica sui concetti chiave, sui benefici e sulle opportunità offerte dall’adozione dei container Linux e di tecnologie open source come Red Hat OpenShift e Gluster.
I punti trattati durante la presentazione sono:
- I container Linux
- Differenze tra virtual machine e container
- Architettura di Red Hat OpenShift
- Container native storage
Per saperne di più, scaricate le slide e guardate il video della presentazione su https://www.par-tec.it/containers-landscape-review
FIWARE Wednesday Webinars - Short Term History within Smart SystemsFIWARE
FIWARE Wednesday Webinar - Short Term History within Smart Systems (2nd April 2020)
Corresponding webinar recording: https://youtu.be/fX_YAc7G4Dk
This webinar will show how to utilise times series components and monitor and display trends within FIWARE applications.
Chapter: Core Context
Difficulty: 3
Audience: Any Technical
Presenter: Jason Fox (Senior Technical Evangelist, FIWARE Foundation)
Functioning incessantly of Data Science Platform with Kubeflow - Albert Lewan...GetInData
Did you like it? Check out our blog to stay up to date: https://getindata.com/blog
The talk is focused on administration, development and monitoring platform with Apache Spark, Apache Flink and Kubeflow in which the monitoring stack is based on Prometheus stack.
Author: Albert Lewandowski
Linkedin: https://www.linkedin.com/in/albert-lewandowski/
___
Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets.
Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries.
https://getindata.com
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
4. DevConf 2019, Brno
4
OPENSHIFT CLUSTER LOGGING
A container log message flow on a node
Kubernetes
master
cri-o Collector
Files
on disk
send off the node
kubelet
kubectl logs <pod>
/var/log/containers/*.log
Container
Container
Container
stdout
stderr
5. DevConf 2019, Brno
5
OPENSHIFT CLUSTER LOGGING
Per Project (Namespace) isolation
Pod
Container
Pod
Container Container
Namespace
Pod
Container Container
Namespace
6. DevConf 2019, Brno
6
OPENSHIFT CLUSTER LOGGING
Openshift-logging Project
Openshift-logging
Namespace
Elasticsearch
Node
Elasticsearch
Node
Elasticsearch
Node
Collector
Collector
Collector
ES service
Kibana
Curator
8. DevConf 2019, Brno
8
THE DATA MODEL
OPERATIONAL &
PERFORMANCE TUNING SKILLED PEOPLE
WHAT ARE THE CHALLENGES
Scaling Elasticsearch with limited resources is hard
10. DevConf 2019, Brno
10
How we store logs into individual indices.
● One index per namespace
● One index per day
CH#1 - THE DATA MODEL
How the data is physically stored is important
Pod
Container
Pod
Container Container
Namespace
Pod
Container Container
Namespace
Today
Today
Today - 1
Today - 1
Today - 2
Today - 2
Today - 3
Today - 3
11. DevConf 2019, Brno
11
Scaling issues:
● Number of indices/shards
● Number of index fields
Reduce number of indices/shards:
● Rollover API
● Commercial (requires license)
● Spread the load to more clusters
CH#1 - THE DATA MODEL
How the data is physically stored is important
Way too
many
shards!
Index
mapping
explosion
13. DevConf 2019, Brno
13
Physical and functional layout:
● Fast local storage
● Dedicated node roles
● Hot-Warm
Monitoring and Alerting:
● Commercial (requires license)
● DIY/3rd party solutions
CH#2 - OPS & PERF TUNING
Topology of Elasticsearch cluster and underlying HW is important
15. DevConf 2019, Brno
15
There is always someone who takes care of
your cluster and typically performs the
following tasks:
● Maintenance
● Upgrades
● Troubleshooting
CH#3 - SKILLED PEOPLE
Someone gets woke up when it happens
17. DevConf 2019, Brno
17
100% open-source and community driven
● Prometheus is systems monitoring
and alerting toolkit
● Pull-based metrics gathering system,
simple text format for metrics
exposition
● PromQL: powerful query language
● CNCF member project
PROMETHEUS
https://prometheus.io/
18. DevConf 2019, Brno
18
Elasticsearch native plugin
● Adding new HTTP endpoint
● Seamless integration with security
existing model (SearchGuard)
● “Low level” Java
● Simple integration testing
● Upgrade require node restart
● https://github.com/vvanholl/elastic
search-prometheus-exporter
ELASTICSEARCH ➔ PROMETHEUS
Elasticsearch does not support metric exposition in Prometheus format
Prometheus exporter
● Requires additional container(s)
● More configuration (security)
● “High level” REST client
● Complex integration testing
● Independent upgrades
20. DevConf 2019, Brno
20
Dashboard to visualize Elasticsearch metrics stored in Prometheus.
● Cluster
● Shrads
● OS & JVM
● Indexing
● Searching
● Caches
● Throttling
● Indices (WIP)
● ...
21. DevConf 2019, Brno
21
ELASTICSEARCH-MIXIN
Magic of Kubernetes-mixin and Jsonnet templates
Bundle of:
● Prometheus recording rules, alerting rules
● Grafana dashboard
Build on top of Jsonnet and Kubernetes-mixin.
Currently at https://github.com/lukas-vlcek/elasticsearch-mixin. This location is likely to
change in the future (community mixins).
23. DevConf 2019, Brno
23
ELASTICSEARCH OPERATOR
Operator pattern for Cluster Logging and Elasticsearch
Elasticsearch
(Cluster)
Kibana
openshift-logging
Namespace
Fluentd
openshift-operators
Namespace
elasticsearch-operator
ClusterLogging
CR
cluster-logging-operator
Elasticsearch
CR
24. DevConf 2019, Brno
24
OPENSHIFT MONITORING INTEGRATION
Prometheus and Grafana artifacts get loaded
openshift-logging
Namespace
openshift-operators
Namespace
openshift-monitoring
Namespace
Elasticsearch-mixin
artifacts Prometheus
Grafana
Alert Manager
25. DevConf 2019, Brno
25
WHAT THE FUTURE HOLDS
Cluster Logging Operator and Elasticsearch Operator are opening new horizons
Operator pattern makes deployment and maintenance of complex components simpler
● Where the logs are routed
● Diversified Elasticsearch Topologies
● Modify cluster based on OpenShift Platform Metrics
28. TRADEMARKS and ATTRIBUTIONS:
Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries. Prometheus is trademark of The Linux Foundation
registered in the United Stated and/or other countries. Red Hat and OpenShift are trademarks or registered trademarks of Red Hat, Inc. or its
subsidiaries in the U.S. and other countries. Grafana is trademarks of Coding Instinct. Used photographs are public domain sourced from
https://www.flickr.com/photos/newcastlelibraries/albums