- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
Prometheus Design and Philosophy by Julius Volz at Docker Distributed System Summit
Prometheus - https://github.com/Prometheus
Liveblogging: http://canopy.mirage.io/Liveblog/MonitoringDDS2016
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaSridhar Kumar N
https://www.youtube.com/playlist?list=PLAiEy9H6ItrKC5PbH7KiELiSEIKv3tuov
-What is Prometheus?
-Difference Between Nagios vs Prometheus
-Architecture
-Alertmanager
-Time series DB
-PromQL (Prometheus Query Language)
-Live Demo
-Grafana
Application Monitoring using Open Source: VictoriaMetrics - ClickHouseVictoriaMetrics
Monitoring is the key to successful operation of any software service, but commercial solutions are complex, expensive, and slow. Let us show you how to build monitoring that is simple, cost-effective, and fast using open source stacks easily accessible to any developer.
We’ll start with the elements of monitoring systems: data ingest, query engine, visualization, and alerting. We’ll then explain and contrast two implementation approaches. The first uses VictoriaMetrics, a fast growing, high performance time series database that uses PromQL for queries. The second is based on ClickHouse, a popular real-time analytics database that speaks SQL. Fast, affordable monitoring is within reach. This webinar provides designs and working code to get you there.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
Prometheus Design and Philosophy by Julius Volz at Docker Distributed System Summit
Prometheus - https://github.com/Prometheus
Liveblogging: http://canopy.mirage.io/Liveblog/MonitoringDDS2016
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaSridhar Kumar N
https://www.youtube.com/playlist?list=PLAiEy9H6ItrKC5PbH7KiELiSEIKv3tuov
-What is Prometheus?
-Difference Between Nagios vs Prometheus
-Architecture
-Alertmanager
-Time series DB
-PromQL (Prometheus Query Language)
-Live Demo
-Grafana
Application Monitoring using Open Source: VictoriaMetrics - ClickHouseVictoriaMetrics
Monitoring is the key to successful operation of any software service, but commercial solutions are complex, expensive, and slow. Let us show you how to build monitoring that is simple, cost-effective, and fast using open source stacks easily accessible to any developer.
We’ll start with the elements of monitoring systems: data ingest, query engine, visualization, and alerting. We’ll then explain and contrast two implementation approaches. The first uses VictoriaMetrics, a fast growing, high performance time series database that uses PromQL for queries. The second is based on ClickHouse, a popular real-time analytics database that speaks SQL. Fast, affordable monitoring is within reach. This webinar provides designs and working code to get you there.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Brian Brazil
Prometheus is a next-generation monitoring system. Since being publicly announced last year it has seen wide-spread interest and adoption. This talk will look at the concepts behind monitoring with Prometheus, and how to use it with Kubernetes which has direct support for Prometheus.
VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...NETWAYS
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
How to monitor your micro-service with Prometheus? How to design metrics, what is USE and RED? Metrics for a REST service with Prometheus, AlertManager, and Grafana.
Grafana Loki: like Prometheus, but for LogsMarco Pracucci
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.
In this talk, we will introduce Loki, its architecture and the design trade-offs in an approachable way. We’ll both cover Loki and Promtail, the agent used to scrape local logs to push to Loki, including the Prometheus-style service discovery used to dynamically discover logs and attach metadata from applications running in a Kubernetes cluster.
Finally, we’ll show how to query logs with Grafana using LogQL - the Loki query language - and the latest Grafana features to easily build dashboards mixing metrics and logs.
VictoriaMetrics 15/12 Meet Up: 2022 Features HighlightsVictoriaMetrics
2022 Features Highlights - Speaker
* MetricsQL Features
- Support for @ modifier
- keep_metric_names modifier
- Advanced label filters’ propagation
- Automatic label filters’ propagation
- Support for short numeric constants
- Distributed query tracing!
- New functions
* vmui Features
- Cardinality explorer!
- Top queries
- Significantly improved usability and stability!
* vmagent Features
- Fetch target response on behalf of vmagent
- Filter targets by url and labels
- /service-discovery page
- Relabel debugging!
- support for absolute _address_
- New service discovery mechanisms
- Multi-tenant support
- Performance improvements
* Relabeling Features
- Conditional relabeling
- Named label placeholders
- Graphite-style relabeling
* vmalert Features
- Better integration with Grafana alerts
- Reusable templates for annotations
- Debugging of alerting rules
- Improved compatibility with Prometheus
* vmctl Features
- Migrate all the data between clusters
- Data migration via Prometheus remote_read protocol
Enterprise Features
- mTLS support
- vmgateway JWT token enhancements
- Automatic restore from backups
- Automatic vmstorage discovery
- Multiple retentions
Various Enhancements
- Environment vars can be referred in command-line flags
- Performance improvements
- Deduplication and downsampling improvements
- Support for metrics push
- Support for Pushgateway data ingestion format
- VictoriaMetrics cluster: multitenancy enhancements
- vmbackup / vmrestore: Azure blob storage
- Official MacOS builds
- Official FreeBSD and OpenBSD builds
- Raspberry PI optimizations
- LTS releases
Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)Brian Brazil
Prometheus is a next-generation monitoring system. Since being publicly announced last year it has seen wide-spread interest and adoption. This talk will look at the concepts behind monitoring with Prometheus, and how to use it with Kubernetes which has direct support for Prometheus.
VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
OSMC 2022 | VictoriaMetrics: scaling to 100 million metrics per second by Ali...NETWAYS
The growth of observability trends and Kubernetes adoption generates more demanding requirements for monitoring systems. Volumes of time series data increase exponentially, and old solutions just can’t keep up with the pace. The talk will cover how and why we created a new open source time series database from scratch. Which architectural decisions, which trade-offs we had to take in order to match the new expectations and handle 100 million metrics per second with VictoriaMetrics. The talk will be interesting for software engineers and DevOps familiar with observability and modern monitoring systems, or for those who’re interested in building scalable high performant databases for time series.
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
How to monitor your micro-service with Prometheus? How to design metrics, what is USE and RED? Metrics for a REST service with Prometheus, AlertManager, and Grafana.
Grafana Loki: like Prometheus, but for LogsMarco Pracucci
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.
In this talk, we will introduce Loki, its architecture and the design trade-offs in an approachable way. We’ll both cover Loki and Promtail, the agent used to scrape local logs to push to Loki, including the Prometheus-style service discovery used to dynamically discover logs and attach metadata from applications running in a Kubernetes cluster.
Finally, we’ll show how to query logs with Grafana using LogQL - the Loki query language - and the latest Grafana features to easily build dashboards mixing metrics and logs.
VictoriaMetrics 15/12 Meet Up: 2022 Features HighlightsVictoriaMetrics
2022 Features Highlights - Speaker
* MetricsQL Features
- Support for @ modifier
- keep_metric_names modifier
- Advanced label filters’ propagation
- Automatic label filters’ propagation
- Support for short numeric constants
- Distributed query tracing!
- New functions
* vmui Features
- Cardinality explorer!
- Top queries
- Significantly improved usability and stability!
* vmagent Features
- Fetch target response on behalf of vmagent
- Filter targets by url and labels
- /service-discovery page
- Relabel debugging!
- support for absolute _address_
- New service discovery mechanisms
- Multi-tenant support
- Performance improvements
* Relabeling Features
- Conditional relabeling
- Named label placeholders
- Graphite-style relabeling
* vmalert Features
- Better integration with Grafana alerts
- Reusable templates for annotations
- Debugging of alerting rules
- Improved compatibility with Prometheus
* vmctl Features
- Migrate all the data between clusters
- Data migration via Prometheus remote_read protocol
Enterprise Features
- mTLS support
- vmgateway JWT token enhancements
- Automatic restore from backups
- Automatic vmstorage discovery
- Multiple retentions
Various Enhancements
- Environment vars can be referred in command-line flags
- Performance improvements
- Deduplication and downsampling improvements
- Support for metrics push
- Support for Pushgateway data ingestion format
- VictoriaMetrics cluster: multitenancy enhancements
- vmbackup / vmrestore: Azure blob storage
- Official MacOS builds
- Official FreeBSD and OpenBSD builds
- Raspberry PI optimizations
- LTS releases
What is the State of my Kafka Streams Application? Unleashing Metrics. | Neil...HostedbyConfluent
"Just as the Apache Kafka Brokers provide JMX metrics to monitor your cluster's health, Kafka Streams provides a rich set of metrics for monitoring your application's health and performance. The metrics to observe for a given use-case of Kafka Streams will vary significantly from application to application. Learning how to build and customize monitoring of those applications will help you maintain a healthy Kafka Streams ecosystem.
Takeaways
* An analysis and overview of the provided metrics, including the new end-to-end metrics of Kafka Streams 2.7.
* See how to extract metrics from your application using existing JMX tooling.
* Walkthrough how to build a dashboard for observing those metrics.
* Explore options of how to add additional JMX resources and Kafka Stream metrics to your application.
* How to verify you built your dashboard correctly by creating a data control set to validate your dashboard.
* Go beyond what you can collect from the Kafka Stream metrics."
Using and Creating SQL Functions with Ammar Hassan Brohi.
String Functions
Numeric Functions
String / Number Conversion Functions
Group Functions
Date and Time Functions
Date Conversion Functions
Creating the PromQL Transpiler for Flux by Julius Volz, Co-Founder | PrometheusInfluxData
Flux is not only a new data scripting and query language — it is also a powerful data processing engine. This talk by Julius Volz will focus on how he worked with the InfluxData team to build PromQL support for the Flux engine. Hear about lessons learned from building the transpiler and recommendations on why and how to use PromQL and Flux. This talk will include a demo and will share the current project progress.
Weave AI Controllers (Weave GitOps Office Hours)Weaveworks
LLMs are one of the rising workloads on Kubernetes and so are the complexities of deploying, managing and fine-tuning them. With this latest extension we can offer a strong blueprint for enterprises on how to keep LLMs OCI contained with the use of Kubernetes, Flux and Weave AI Controllers.
The Highlights:
* Simplified deployment, management, and fine-tuning of LLMs on any Kubernetes infrastructure.
* Strong security and governance ensured through GitOps workflows and a robust signing and verification process.
The Whys:
* Security, Governance & Compliance: Ensures vulnerability-free and compliant deployments.
* Seamless Integration: Works with existing systems, including Red Hat OpenShift.
* GitOps for Productivity & Collaboration: Leverages the power of Flux and Kubernetes for automated, streamlined workflows.
The Weave AI Controllers are an out of the box extension for Flux and are shipped and supported with Weave GitOps Assured (https://www.weave.works/product/gitops) and Enterprise (https://www.weave.works/product/gitops-enterprise/).
Read our latest blog for more information (https://www.weave.works/blog/weave-ai-controllers) and visit GitHub to get started - https://github.com/weave-ai/weave-ai
Flamingo: Expand ArgoCD with Flux (Office Hours)Weaveworks
Flamingo is an open source tool that allows for integrated use of both Flux and ArgoCD, the two leading GitOps solutions available today.
* See how to integrate the two most used CNCF projects together to create flexible and extensible GitOps solutions.
* Learn how to use Flux’s powerful and secure controllers with ArgoCD’s web-based GUI.
* Understand how Flamingo provides a path towards Platform Engineering for ArgoCD users.
* Explore extending ArgoCD to manage Infrastructure as Code through Flux’s Terraform Controller.
For more information visit: https://github.com/flux-subsystem-argo/flamingo
Webinar: Capabilities, Confidence and Community – What Flux GA Means for YouWeaveworks
Flux, the original GitOps project, began its development in a small London office back in 2017 with the goal to bring continuous delivery (CD) to developers, platform and cluster operators working with Kubernetes. From donating the project to the CNCF, its continued growth within the cloud native community, to its achievement of passing rigorous battle tests for security, longevity and governance, it’s little wonder that Flux v2 has reached yet another celebratory milestone – General Availability (GA).
Flux is the GitOps platform of choice for many enterprise companies such as SAP, Volvo Cars, and Axel Springer; and is embedded within AKS, Azure Arc and EKS Anywhere. It provides extensive automation to CI/CD, security and audit trails, and reliability through canary deployments and rollback capabilities.
Join this webinar by Flux maintainers and creators and discover:
* Latest release features and roadmap for the future.
* Interesting use cases for Flux (e.g security).
* Flux capabilities you may not be aware of (e.g. extensions).
* Joining the vibrant Flux community.
* How to leverage Flux in a supported enterprise environment today.
Although not an entirely new concept, Platform Engineering and Internal Developer Platforms (IDPs) are all the rage due to their potential to increase development velocity and deployment frequency while boosting reliability and security.
Join Joe Dahlquist, VP of PMM and Mohamed Ahmed, VP of Developer Platforms at Weaveworks to learn the 6 tell-tale signs your company should implement a platform engineering approach. The webinar draws on hundreds of conversations with SRE’s, developers, and platform engineering teams to help you better understand what works, what doesn’t and what might be missing from your strategy. Attendees can apply these learnings to their first (or next) developer platform regardless of your build vs. buy journey.
You will learn:
* The difference between Internal Developer Platforms and Platform Engineering
* Why platform engineering now?
* How Dev and Ops benefit from an IDP
* 6 tell-tale signs to start platform engineering
* Drafting your platform engineering strategy - where to begin and what to avoid
SRE and GitOps for Building Robust Kubernetes Platforms.pdfWeaveworks
In today's technology-driven landscape, ensuring the reliability and stability of systems is critical for organizations to deliver exceptional user experiences. Site Reliability Engineering (SRE) has emerged as a proven methodology to achieve operational excellence and elevate performance.
By combining SRE and GitOps, organizations can leverage the benefits of both methodologies. GitOps provides a reliable and auditable approach to managing infrastructure and application changes, ensuring that all deployments are version-controlled and consistent across environments. This aligns with the SRE principle of implementing standardized and automated processes for maintaining system reliability.
Join our live webinar as we introduce the fundamentals and significance of SRE and GitOps, and provide actionable strategies for implementation. We’ll also explore the features of Weave GitOps that integrate SRE and GitOps practices to streamline workflows to support system reliability and stability.
You will learn:
An overview and correlation of key SRE and GitOps best practices
The 5 keys DORA metrics for measuring performance of software delivery.
How to leverage continuous delivery and progressive delivery to enhance application stability.
How Weave GitOps can reliably simplify the management of infrastructure and applications, with real-world customer examples illustrating their impact.
Webinar: End to End Security & Operations with Chainguard and Weave GitOpsWeaveworks
One of the key values of GitOps relies on its fully declarative single source of truth in Git for the desired state of your entire system – configuration that continuously reconciles with the runtime of the system.
Validating committer identity in your Git repository is a critical component towards a secure GitOps solution. Although basic capabilities are provided by Git service providers, more granular controls for governance and compliance are a requirement to satisfy most enterprise grade implementations.
How do you keep that end to end process secure, from Git to Runtime?
Join Weaveworks and Chainguard for a live webinar where we will look at how Chainguard Enforce for Git together with Weave GitOps Enterprise Policy Engine allows you to secure your end to end GitOps workflows, from Git to Runtime.
You will learn how to:
- Use Chainguard Enforce for Git to ensure only authorized GitOps tooling can modify your desired state.
- Provide a secure identity to Weave GitOps Enterprise for all Git operations.
- Use Weave GitOps Policy Engine to guarantee compliance on admission.
Flux Beyond Git Harnessing the Power of OCIWeaveworks
Watch the recap: https://youtu.be/gKR95Kmc5ac
In this KubeCon Europe 2023 session, Stefan and Hidde will talk about the latest developments of Flux around the Open Container Initiative (OCI). The focus will be on how OCI can serve as the single source of truth for both application code (container images) and configuration (OCI artifacts). We will start by explaining how Flux can be used as a package manager for distributing Kubernetes configs and Terraform modules as OCI artifacts. Afterwards, we will demonstrate how to build a secure delivery pipeline that leverages Flux integrations with GitHub Actions and keyless signatures from Sigstore Cosign. Lastly, we will touch upon the upcoming plans for 2023 and the significance of OCI in the future of continuous delivery with Flux.
Automated Provisioning, Management & Cost Control for Kubernetes ClustersWeaveworks
In today’s economic climate, IT departments are feeling the pressure to reduce costs which can have a significant effect on development teams, and more specifically, Kubernetes strategies. For many organizations, there is a good chance that many Kubernetes resources are overprovisioned, and it’s often difficult to visualize which processes are responsible for this unnecessary spend.
Weaveworks has joined forces with KubeCost to show you how to “do more with less” by easily integrating a Kubernetes FinOps solution into your existing workflows and seamlessly automating the provisioning and management of FinOps enabled Kubernetes clusters from a single UI / dashboard.
Join this webinar to discover best practices for monitoring and reducing Kubernetes spend, while balancing cost, performance, and reliability.
What you’ll learn:
- Best practices for implementing a FinOps strategy in your organization.
- Cluster management and templating capabilities using Weave GitOps for automating FinOps.
- How to use predefined, automated policies for reliable cost control across your Kubernetes environment.
How to Avoid Kubernetes Multi-tenancy CatastrophesWeaveworks
Picture this… It’s the middle of the night on a Saturday, and the sound of slack messages rolling in rouses you from slumber. Then two text messages chime in quick succession. As you grab your phone and pry open an eye to figure out WTF, the phone rings - and it’s your boss!? You stammer out a “Hello?”
She sounds alarmed. “Wake up, we have a big problem”
“It’s two-in-the-morning, what problem?” you croak back.
“I guess you missed the alerts while you were sleeping…API endpoints in prod are getting knocked over, and the tokens responsible are yours.”
“They’re what? How?”
“Get to your machine and jump on the meeting link I just sent - everybody’s waiting”
Yikes. Join Weaveworks for some real-world tales from the trenches, and learn about the 5 simple things you can do to prevent making a royal mess of Tenancy in Kubernetes. Hear from developers that got that late night call because of a bone-headed accident, and teams affected by gob-smacking access and permissions foul-ups. Luckily for us, they were happy to tell us the tales so we can learn from their pain.
Weave GitOps Workspaces is a new feature that enables multi-tenancy so platform engineers can scale their GitOps workflows across numerous development teams. Oh yeah, it also wards -off wake-up calls in the middle of the night, which is nice.
Watch this webinar recording to learn:
- How Weave GitOps simplifies tenancy management
- How security guardrails keep you from blowing a hole in your app, and across your team
- 5 takeaways for enabling Kubernetes tenancy safely and effectively for your teams
Building internal developer platform with EKS and GitOpsWeaveworks
An internal developer platform (IDP) is a set of standardized tools and technologies that enables development teams to self-service, offering convenient access to resources they need to create and deploy compliant code. The ultimate goal is to facilitate automation, autonomy and productivity across large teams. However, creating an IDP is highly complex, especially when bridging hybrid scenarios. In fact, build timelines can take anywhere between one to two years!
In this Techstrong Learning Experience, we will discuss how platform engineers can more efficiently build an IDP with Amazon EKS and Weave GitOps and accelerate cloud-native adoption while speeding up migration of existing applications to the cloud.
Our experts will also introduce EKS Blueprints, a collection of infrastructure-as-code (IaC) modules like Terraform and AWS Cloud Development Kit (AWS CDK) that will help you configure and deploy consistent EKS clusters across on-premises and cloud.
Key Takeaways:
- Why you should build a self-service IDP
- How to leverage EKS, GitOps and EKS Blueprints to build your IDP
- A review of use cases and benefits of an IDP
GitOps Testing in Kubernetes with Flux and Testkube.pdfWeaveworks
GitOps is amazing... until you can't apply it! This has been the case mostly for testing where it continues to be more of a push than a pull in organizations' DevOps pipelines.
Join us in this talk to learn the benefits of improving your existing testing pipeline with Testkube, an open source project that brings tests inside your Kubernetes cluster, and FluxCD adding the GitOps sprinkles to testing!
Speaker: Abdallah Abedraba, Product Leader at Testkube
Abdallah works at Testkube, a Kubernetes native testing framework. In his prior experiences, he has tried everything from software engineering to product management, and now working as a Developer Advocate, on open source (a dream of his!) evangelizing all things Testing and Kubernetes. In his free time, he enjoys attending developer conferences and meetups, as well as spending time at the movies and actively listening to music.
Intro to GitOps with Weave GitOps, Flagger and LinkerdWeaveworks
You may not think of "GitOps" and "service mesh" together – but maybe you should! These two wildly different technologies are each enormously capable independently, and combined they deliver far more than the sum of their parts: a single Git commit can control workflows customized for your exact situation by taking advantage of the service mesh's ability to measure and manipulate traffic anywhere in your application's call graph, and you can rest easy knowing that Git is preserving the complete configuration for your entire application every step of the way.
See how these technologies can work together to tackle complex problems in cloud-native applications.
What you’ll get out of this:
* Understand what GitOps and service meshes can - and can't - do for you.
* Understand basic operations with GitOps and Linkerd.
* Understand the basics of continuous deployment with Weave GitOps and Linkerd.
Implementing Flux for Scale with Soft Multi-tenancyWeaveworks
Soft multi-tenancy can be hard to achieve and secure. Multiple tenants sharing the same cluster means there are global objects, like Custom Resource Definitions (CRDs), namespaces, and so on, that you don’t want tenants controlling. Platform admins, cluster admins, and tenants, should be separated, with dedicated namespaces, role bindings, node groups, taints and tolerations, etc.
With Flux, tenant isolation is enforced by default, so you don’t have to worry about accidental tenant cross-over / cross-contamination.
In this session, Priyanka “Pinky” Ravi, Developer Experience Engineer at Weaveworks, will walk you through how to set up multi-tenancy on an existing Kubernetes cluster and manage several tenants within the cluster.
Take advantage of the benefits that come with infrastructure as code.
Accelerating Hybrid Multistage Delivery with Weave GitOps on EKSWeaveworks
Join Leo Murillo, Principal Solutions Architect at Weaveworks and Rama Ponnuswami, Sr. Container Specialist at AWS, as they walk through accelerating Multi-stage delivery on GitOps. If you already have EKS-A, you are ready to automate the release of multistage delivery. Thus, allowing you to deploy more often and reliably with less overhead.
In this Webinar, we cover:
- Best practices for CI/CD, GitOps and Application Pipeline Management.
- Simple cluster management across Kubernetes hybrid infrastructure.
- Multistage deployments using Weave GitOps for EKS and EKS-A using a single UI dashboard.
Shift Deployment Security Left with Weave GitOps & Upbound’s Universal Crossp...Weaveworks
In this session, we’ve partnered with Upbound to showcase how to effectively manage application delivery while maintaining a high level of security using Weave GitOps and Upbound. Managing a stateful application deployment with a relational database, Weave GitOps can recognize if there is a policy violation and correct it before deploying the application.
Join us as we demonstrate the scenarios where:
All changes to application configuration are managed through Git workflows
Upbound’s Universal Crossplane allows you to build, deploy, and manage your cloud platforms
GitOps provides an extra layer of security by removing the need for direct access to Kubernetes clusters
Policy-as-Code guarantees security, resilience and coding standards compliance
Watch the recording: xx
Securing Your App Deployments with Tunnels, OIDC, RBAC, and Progressive Deliv...Weaveworks
In a joint webinar with Traefik Labs, we show how Traefik Hub, a SaaS-based cloud native networking platform, helps you publish your containers securely in seconds with tunnels, OIDC authentication and automated TLS certificate management. And, how you can combine that with Weave GitOps to achieve continuous application delivery using progressive delivery strategies for risk-free and reliable deployments.
Security is key, so we showcase multi-tenancy for full RBAC across the different deployment stages, and trusted delivery best practices for continuous security and compliance baked in.
Learn how:
- To utilize canary deployments for reliable and risk-free application deployments.
- GitOps lets you automate and secure the publishing of containers at the edge consistently.
- Easy it is to deploy, update and manage your application workloads on Kubernetes.
- To publish containers securely using tunnels, OIDC authentication and TLS certificate management.
Flux’s Security & Scalability with OCI & Helm Slides.pdfWeaveworks
During this session Kingdon Barrett, OSS Engineer at Weaveworks & Flux Maintainer, will show you how to quickly create scalable and Cosign-verified GitOps configurations with Flux using the same process with two demo environments: one will be a Kustomize Environment and the other a Helm-based environment.
Flux Security & Scalability using VS Code GitOps Extension Weaveworks
Recently Flux has released two new features (OCI and Cosign) for scalable and secure GitOps. Juozas Gaigalas, a Developer Experience Engineer at Weaveworks, will demonstrate how developers and platform engineers can quickly create scalable and Cosign-verified GitOps configurations using VS Code GitOps Tools extension. New and experienced Flux users can learn about Flux’s OCI and Cosign support through this demo.
Deploying secure, cloud native stateful applications requires a high level of performance across hybrid and multi-cloud environments.
Using the scalable, highly performant storage provided by Ondat in combination with Weave GitOps Trusted Delivery, you can shift left security and accelerate software development.
Watch this on-demand webinar as we demonstrate how:
- All changes to application configuration are managed through Git workflows
GitOps provides an extra layer of security by removing the need for direct access to Kubernetes clusters.
- Policy-as-Code guarantees security, resilience and coding standards compliance.
- To dynamically provision highly available persistent volumes by simply deploying Ondat anywhere with a simple operator profile.
- All data services such as replication, compression and encryption, are optimized and accelerated to scale on any platform with Ondat’s low latency data plane.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
2. Agenda
1. What is PromQL
2. PromQL Operators
3. PromQL Functions
4. Hands On: Building Queries in PromQL
5. Hands On: Visualizing PromQL in Grafana
6. Training on real app
7. What’s next?
3. What is PromQL
• Powerful Query Language of Prometheus
• Provides built in operators and functions
• Vector-based calculations like Excel
• Expressions over time-series vectors
4. Expressions (and sub-expressions)
• Instant Vector - set of time series containing single sample for each time series, all sharing
same timestamp
• e.g. http_request_count => results in:
• http_request_count{status=“200”} 20
• http_request_count{status=“404”} 3
• http_request_count{status=“500”} 5
5. Expressions (and sub-expressions)
• Instant Vector - set of time series containing single sample for each time series, all sharing
same timestamp
• e.g. http_request_count => results in:
• http_request_count{status=“200”} 20
• http_request_count{status=“404”} 3
• http_request_count{status=“500”} 5
• Range Vector - set of time series containing a range of data points over time for each series
• e.g. http_request_count[5m] => results in:
• http_request_count{status=“200”}
6. Expressions (and sub-expressions)
• Instant Vector - set of time series containing single sample for each time series, all sharing
same timestamp
• e.g. http_request_count => results in:
• http_request_count{status=“200”} 20
• http_request_count{status=“404”} 3
• http_request_count{status=“500”} 5
• Range Vector - set of time series containing a range of data points over time for each series
• e.g. http_request_count[5m] => results in:
• http_request_count{status=“200”}
• Scalar - as a literal and as result of an expression
• String - only currently as a literal in an expression
12. Operators: Binary
• Arithmetic: +, -, *, /, %, ^
– scalar/scalar
– vector/scalar
– vector/vector
• Comparison: ==, !=, >, <, >=, <=
– filters results unless bool operator provided (converts 0 or 1)
– scalar/scalar requires bool operator
– vector/scalar & vector/vector drops elements unless bool operator provided
• Logical/Set Binary: only defined between Instant Vectors
– and = intersection between vector1 and vector2
– or = union of vector1 and vector2
– unless = elements of vector1 for which no matches in vector2
14. Operators: Vector Matching
• Label Matching
– ignoring keyword
– on keyword
• One-to-one - finds unique pair of entries with all labels
matching
15. Operators: Vector Matching
• Label Matching
– ignoring keyword
– on keyword
• One-to-one - finds unique pair of entries with all labels
matching
• Many-to-one / One-to-many - where each element on a “one”
side can multiple elements on the “many” side
– group_left v group_right determines cardinality
– only used for comparison and arithmetic operations
16. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
17. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector
18. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector
• by clause drops labels not listed from the resulting vector
19. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector
• by clause drops labels not listed from the resulting vector
• keep_common (with by) will keep labels that exist in all elements but not listed in the by clause
20. Operators: Aggregation
• Aggregate elements of a single Instant Vector resulting in a new vector of fewer elements w/ aggregated values
– sum, avg
– min, max
– stddev, stdvar
– count, count_values*
– bottomk*, topk*
– quantile*
*takes a parameter before the vector
• without clause removes listed labels from resulting vector
• by clause drops labels not listed from the resulting vector
• keep_common (with by) will keep labels that exist in all elements but not listed in the by clause
• topk/bottomk - only subset of original values are returned including original labels - by and without only bucket
the input
21. Functions: Utilities
• time() - number of seconds since Unix
Epoch when the expression is run
• vector(s scalar) - returns a vector
from a scalar
• scalar(v vector) - returns scalar
value of a single sampled vector or NaN
24. Functions: Range Vector
• changes()ˆ
• delta()˚*, idelta()˚* - diff between first and last in each time series element
• deriv()* - per sec derivative using simple linear regression
• holt_winters(v, sf scalar, tf scalar)* - smooth value for time series based on
range in v
• increase()ˆ - syntactic sugar for rate(v[T]) * (seconds in T)
• irate()ˆ, rate()ˆ - per second instant/avg rate of increase
• predict_linear(v, t scalar)* - predict value at time t using simple linear regression
• resets()ˆ - number of times a counter reset
• <aggregation>_over_time()˚ - aggregate each series of a range vector over time
returning instant vector with per series aggregation results
• ˚returns an instant vector, *should only be used with gauges, ˆshould only be used with
counters
26. Metrics Types - Basic Counters
• counter - single numeric metric that only
goes up
• gauge - single numeric metric that
arbitrarily goes up or down
27. Metric Types - Sampling Counters
• histogram - samples observations and
counts them in configurable buckets
• summary - samples observations and
counts them
28. Metric Types - Sampling Counters
• histogram - samples observations and
counts them in configurable buckets
• summary - samples observations and
counts them
30. Metric Types - Sampling Counters
• both histogram and summary have:
– <name>_sum - time series summing the value of all observations
– <name>_count - time series counter for the number of observations taken
31. Metric Types - Sampling Counters
• both histogram and summary have:
– <name>_sum - time series summing the value of all observations
– <name>_count - time series counter for the number of observations taken
• histograms:
– buckets are configured on client when creating metrics
– time series for each bucket as <name>_bucket{…,le=“<bucket-upper-bound>”} counting the number
of observations less than or equal to the upper bound of the bucket
– ad-hoc quantile specification using the histogram_quantile(quantile, instant-vector) function
32. Metric Types - Sampling Counters
• both histogram and summary have:
– <name>_sum - time series summing the value of all observations
– <name>_count - time series counter for the number of observations taken
• histograms:
– buckets are configured on client when creating metrics
– time series for each bucket as <name>_bucket{…,le=“<bucket-upper-bound>”} counting the number
of observations less than or equal to the upper bound of the bucket
– ad-hoc quantile specification using the histogram_quantile(quantile, instant-vector) function
• summaries:
– quantiles are defined on the client when creating metrics
– time series for each quantile as <name>{…,quantile=“<quantile-upper-bound>”} keeping the
streaming quantile calculation from the client
– are generally not aggregatable
39. RED Monitoring
• (Request) Rate - the number of requests
per second your services are serving
• (Request) Errors - the number of failed
requests per second
• (Request) Duration - distributions of the
amount of time each request takes
41. Join the Weave user group!
meetup.com/pro/Weave/
weave.works/help
42. Other topics
• Kubernetes 101
• Continuous delivery: hooking up my CI/CD
pipeline to Kubernetes
• Network policy for security
We have talks on all these topics in the Weave
user group!
43. Thanks! Questions?
We are hiring!
DX in San Francisco
Engineers in London & SF
weave.works/weave-company/hiring