The adoption and popularity of the microservices architecture continues to grow across a spectrum of enterprises in every industry. Although a consensus on an implementation standard has yet to be reached, advanced design patterns and lessons learned about the complexities and pitfalls of deploying microservices at scale have been established by thought leaders and the development community. With Redis and Kafka becoming de facto standards across most microservices architectures, we will discuss how their combination can be used to simplify the implementation of event-driven design patterns that will provide real-time performance, scalability, resiliency, traceability to ensure compliance, observability, reduced technology sprawl, and scale to thousands of services. In this discussion, we will decompose a real-time event-driven payment-processing microservices workflow to explore capturing telemetry data, event sourcing, CQRS, orchestrated SAGA workflows, inter-service communication, state machines, and more.
Microservices architectures are changing the way that organizations build their applications and infrastructure. Companies can now achieve new levels of scale and efficiency by disaggregating their large, monolithic applications into small, independent “micro services”, each of which perform different functions. In this session, we’ll introduce the concept of microservices, help you evaluate whether your organization is ready for microservices, and discuss methods for implementing these architectures. We’ll also cover topics such as using API gateways, enabling self-service infrastructure provisioning, and ways to manage your microservices.
Learn all about microservices from Product Marketing Manager Dan Giordano. We'll cover how to get started, the benefits, potential challenges, and how SmartBear can help.
Redis and Kafka - Simplifying Advanced Design Patterns within Microservices A...HostedbyConfluent
The adoption and popularity of the microservices architecture continues to grow across a spectrum of enterprises in every industry. Although a consensus on an implementation standard has yet to be reached, advanced design patterns and lessons learned about the complexities and pitfalls of deploying microservices at scale have been established by thought leaders and the development community. With Redis and Kafka becoming de facto standards across most microservices architectures, we will discuss how their combination can be used to simplify the implementation of event-driven design patterns that will provide real-time performance, scalability, resiliency, traceability to ensure compliance, observability, reduced technology sprawl, and scale to thousands of services. In this discussion, we will decompose a real-time event-driven payment-processing microservices workflow to explore capturing telemetry data, event sourcing, CQRS, orchestrated SAGA workflows, inter-service communication, state machines, and more.
OSMC 2022 | Ignite: Observability with Grafana & Prometheus for Kafka on Kube...NETWAYS
Self-managing a highly scalable distributed system with Apache Kafka® at its core is not an easy feat. That’s why operators prefer tooling such as Confluent Control Center for administering and monitoring their deployments. However, sometimes, you might also like to import monitoring data into a third-party metrics aggregation platform for service correlations, consolidated dashboards, root cause analysis, or more fine-grained alerts. If you’ve ever asked a question along these lines: Can I export JMX data from Confluent clusters to my monitoring system with minimal configuration? What if I could correlate this service’s data spike with metrics from Confluent clusters in a single UI pane? Can I configure some Grafana dashboards for Confluent clusters?
This talk will enable you on achieving the below:
Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana (this article)
Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients
Microservices architectures are changing the way that organizations build their applications and infrastructure. Companies can now achieve new levels of scale and efficiency by disaggregating their large, monolithic applications into small, independent “micro services”, each of which perform different functions. In this session, we’ll introduce the concept of microservices, help you evaluate whether your organization is ready for microservices, and discuss methods for implementing these architectures. We’ll also cover topics such as using API gateways, enabling self-service infrastructure provisioning, and ways to manage your microservices.
Learn all about microservices from Product Marketing Manager Dan Giordano. We'll cover how to get started, the benefits, potential challenges, and how SmartBear can help.
Redis and Kafka - Simplifying Advanced Design Patterns within Microservices A...HostedbyConfluent
The adoption and popularity of the microservices architecture continues to grow across a spectrum of enterprises in every industry. Although a consensus on an implementation standard has yet to be reached, advanced design patterns and lessons learned about the complexities and pitfalls of deploying microservices at scale have been established by thought leaders and the development community. With Redis and Kafka becoming de facto standards across most microservices architectures, we will discuss how their combination can be used to simplify the implementation of event-driven design patterns that will provide real-time performance, scalability, resiliency, traceability to ensure compliance, observability, reduced technology sprawl, and scale to thousands of services. In this discussion, we will decompose a real-time event-driven payment-processing microservices workflow to explore capturing telemetry data, event sourcing, CQRS, orchestrated SAGA workflows, inter-service communication, state machines, and more.
OSMC 2022 | Ignite: Observability with Grafana & Prometheus for Kafka on Kube...NETWAYS
Self-managing a highly scalable distributed system with Apache Kafka® at its core is not an easy feat. That’s why operators prefer tooling such as Confluent Control Center for administering and monitoring their deployments. However, sometimes, you might also like to import monitoring data into a third-party metrics aggregation platform for service correlations, consolidated dashboards, root cause analysis, or more fine-grained alerts. If you’ve ever asked a question along these lines: Can I export JMX data from Confluent clusters to my monitoring system with minimal configuration? What if I could correlate this service’s data spike with metrics from Confluent clusters in a single UI pane? Can I configure some Grafana dashboards for Confluent clusters?
This talk will enable you on achieving the below:
Monitoring Your Event Streams: Integrating Confluent with Prometheus and Grafana (this article)
Monitoring Your Event Streams: Tutorial for Observability Into Apache Kafka Clients
This deck is about Microservices Architecture and why do we need it, architecture patterns which need to be followed during Microservices development, and about few tricky questions like API Versioning and
Decomposition Recipes
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments.
This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.
A presentation to explain the microservices architecture, the pro and the cons, with a view on how to migrate from a monolith to a SOA architecture. Also, we'll show the benefits of the microservices architecture also for the frontend side with the microfrontend architecture.
Developing Event-driven Microservices with Event Sourcing & CQRS (gotoams)Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC.
In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
Druid provides sub-second query latency and Flink provides SQL on streams allowing rich transformation/enrichment of events as it happens. In this talk we will learn how Lyft
uses flink sql and druid together to support real time analytics.
Meetup: https://www.meetup.com/druidio/events/252515792/
InfluxDB 101 - Concepts and Architecture | Michael DeSa | InfluxDataInfluxData
Complete introduction to time series, the components of InfluxDB, how to get started, and how to think of your metrics problems with the InfluxDB platform in mind. What is a tag, and what is a value? Come and find out!
NATS & Docker Meetup in Toronto - August 2016
Implementing Microservices with NATS, Diogo Monteiro
-How Aytra uses NATS
-Benefits of using NATS for inter service communication
-Lessons learned adopting NATS
-Overview of Houston NATS library
-Demo of Aytra
You can learn more about NATS at http://www.nats.io
VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
PromQL Deep Dive - The Prometheus Query Language Weaveworks
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxRomanKhavronenko
VictoriaMetrics and Grafana Mimir are time series databases with support of mostly the same protocols and APIs. However, they have different architectures and components, which makes the comparison more complicated. In the talk, we'll go through the details of the benchmark where I compared both solutions. We'll see how VictoriaMetrics and Mimir are dealing with identical workloads and how efficient they’re with using the allocated resources.
The talk will cover design and architectural details, weak and strong points, trade-offs, and maintenance complexity of both solutions.
Spring Framework Petclinic sample applicationAntoine Rey
Spring Petclinic is a sample application that has been designed to show how the Spring Framework can be used to build simple but powerful database-oriented applications.
The fork named Spring Framework Petclinic maintains a version both with a plain old Spring Framework configuration and a 3-layer architecture (i.e. presentation --> service --> repository).
[WSO2Con EU 2017] Microservices for EnterprisesWSO2
Microservice architecture (MSA) is fast becoming a popular architecture pattern in today’s agile enterprises. Its iterative architecture and development methodologies are attracting the interest of architects who need continuous delivery to fulfill business needs. But, is every characteristic of MSA new or even pragmatic? Can MSA alone help you solve your enterprise challenges? This session will explore how middleware plays a key role in successful MSA-based implementations.
This deck is about Microservices Architecture and why do we need it, architecture patterns which need to be followed during Microservices development, and about few tricky questions like API Versioning and
Decomposition Recipes
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments.
This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.
A presentation to explain the microservices architecture, the pro and the cons, with a view on how to migrate from a monolith to a SOA architecture. Also, we'll show the benefits of the microservices architecture also for the frontend side with the microfrontend architecture.
Developing Event-driven Microservices with Event Sourcing & CQRS (gotoams)Chris Richardson
Modern, cloud-native applications typically use a microservices architecture in conjunction with NoSQL and/or sharded relational databases. However, in order to successfully use this approach you need to solve some distributed data management problems including how to maintain consistency between multiple databases without using 2PC.
In this talk you will learn more about these issues and how to solve them by using an event-driven architecture. We will describe how event sourcing and Command Query Responsibility Separation (CQRS) are a great way to realize an event-driven architecture. You will learn about a simple yet powerful approach for building, modern, scalable applications.
Druid provides sub-second query latency and Flink provides SQL on streams allowing rich transformation/enrichment of events as it happens. In this talk we will learn how Lyft
uses flink sql and druid together to support real time analytics.
Meetup: https://www.meetup.com/druidio/events/252515792/
InfluxDB 101 - Concepts and Architecture | Michael DeSa | InfluxDataInfluxData
Complete introduction to time series, the components of InfluxDB, how to get started, and how to think of your metrics problems with the InfluxDB platform in mind. What is a tag, and what is a value? Come and find out!
NATS & Docker Meetup in Toronto - August 2016
Implementing Microservices with NATS, Diogo Monteiro
-How Aytra uses NATS
-Benefits of using NATS for inter service communication
-Lessons learned adopting NATS
-Overview of Houston NATS library
-Demo of Aytra
You can learn more about NATS at http://www.nats.io
VictoriaLogs: Open Source Log Management System - PreviewVictoriaMetrics
VictoriaLogs Preview - Aliaksandr Valialkin
* Existing open source log management systems
- ELK (ElasticSearch) stack: Pros & Cons
- Grafana Loki: Pros & Cons
* What is VictoriaLogs
- Open source log management system from VictoriaMetrics
- Easy to setup and operate
- Scales vertically and horizontally
- Optimized for low resource usage (CPU, RAM, disk space)
- Accepts data from Logstash and Fluentbit in Elasticsearch format
- Accepts data from Promtail in Loki format
- Supports stream concept from Loki
- Provides easy to use yet powerful query language - LogsQL
* LogsQL Examples
- Search by time
- Full-text search
- Combining search queries
- Searching arbitrary labels
* Log Streams
- What is a log stream?
- LogsQL examples: querying log streams
- Stream labels vs log labels
* LogsQL: stats over access logs
* VictoriaLogs: CLI Integration
* VictoriaLogs Recap
PromQL Deep Dive - The Prometheus Query Language Weaveworks
- What is PromQL
- PromQL operators
- PromQL functions
- Hands on: Building queries in PromQL
- Hands on: Visualizing PromQL in Grafana
- Prometheus alerts in PromQL
- Hands on: Creating an alert in Prometheus with PromQL
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxRomanKhavronenko
VictoriaMetrics and Grafana Mimir are time series databases with support of mostly the same protocols and APIs. However, they have different architectures and components, which makes the comparison more complicated. In the talk, we'll go through the details of the benchmark where I compared both solutions. We'll see how VictoriaMetrics and Mimir are dealing with identical workloads and how efficient they’re with using the allocated resources.
The talk will cover design and architectural details, weak and strong points, trade-offs, and maintenance complexity of both solutions.
Spring Framework Petclinic sample applicationAntoine Rey
Spring Petclinic is a sample application that has been designed to show how the Spring Framework can be used to build simple but powerful database-oriented applications.
The fork named Spring Framework Petclinic maintains a version both with a plain old Spring Framework configuration and a 3-layer architecture (i.e. presentation --> service --> repository).
[WSO2Con EU 2017] Microservices for EnterprisesWSO2
Microservice architecture (MSA) is fast becoming a popular architecture pattern in today’s agile enterprises. Its iterative architecture and development methodologies are attracting the interest of architects who need continuous delivery to fulfill business needs. But, is every characteristic of MSA new or even pragmatic? Can MSA alone help you solve your enterprise challenges? This session will explore how middleware plays a key role in successful MSA-based implementations.
Deep-dive into Microservices Patterns with Replication and Stream Analytics
Target Audience: Microservices and Data Architects
This is an informational presentation about microservices event patterns, GoldenGate event replication, and event stream processing with Oracle Stream Analytics. This session will discuss some of the challenges of working with data in a microservices architecture (MA), and how the emerging concept of a “Data Mesh” can go hand-in-hand to improve microservices-based data management patterns. You may have already heard about common microservices patterns like CQRS, Saga, Event Sourcing and Transaction Outbox; we’ll share how GoldenGate can simplify these patterns while also bringing stronger data consistency to your microservice integrations. We will also discuss how complex event processing (CEP) and stream processing can be used with event-driven MA for operational and analytical use cases.
Business pressures for modernization and digital transformation drive demand for rapid, flexible DevOps, which microservices address, but also for data-driven Analytics, Machine Learning and Data Lakes which is where data management tech really shines. Join us for this presentation where we take a deep look at the intersection of microservice design patterns and modern data integration tech.
Building Cloud-Native App Series - Part 5 of 11
Microservices Architecture Series
Microservices Architecture,
Monolith Migration Patterns
- Strangler Fig
- Change Data Capture
- Split Table
Infrastructure Design Patterns
- API Gateway
- Service Discovery
- Load Balancer
Stateful on Stateless - The Future of Applications in the CloudMarkus Eisele
Most developers building applications on top of Kubernetes are still mainly relying on stateless protocols and design. The problem is that focusing exclusively on a stateless design ignores the hardest part in distributed systems: managing state—your data.
The challenge is not designing and implementing the services themselves, but managing the space in between the services: data consistency guarantees, reliable communication, data replication and failover, component failure detection and recovery, sharding, routing, consensus algorithms and so on.
Kubernetes and Akka work well together since each being responsible for a different layer and function in the application stack. Kubernetes allows for coarse-grained container-level management of resilience and scalability. Akka allows for fine-grained entity-level management of resilience and scalability. This talk demonstrates how the two play together to deliver the future of stateful applications in the cloud.
Microservices: Where do they fit within a rapidly evolving integration archit...Kim Clark
Do microservices force us to look differently at the way we lay down and evolve our integration architecture, or are they purely about how we build applications? Are microservices a new concept, or an evolution of the many ideas that came before them? What is the relationship between microservices and other key initiatives such as APIs, SOA, and Agile. In this session, we will unpick what microservices really are, and indeed what they are not. We will consider whether there is something unique about this particular point time in technology that has enables microservice concepts to take hold. Finally, we will look at if, when, where and how an enterprise can take on the benefits of microservices, and what products and technologies are applicable for that journey.
The introduction covers the following
1. What are Microservices and why should be use this paradigm?
2. 12 factor apps and how Microservices make it easier to create them
3. Characteristics of Microservices
Note: Please download the slides to view animations.
Microservices design principles establish some standard practices for planning, developing, and implementing a distributed architecture for your application. Read about some of the most common characteristics of design principles, its examples, and implementations carried out by various companies worldwide.
Architecture principles, How to, Patterns,
Comparison with other SOA styles
Pragmatic options to scale Monoliths
Illustrated with the Netflix stack and Gilt, SoundCloud testimonials
To go futher, check 200 - Building Microservices
http://fr.slideshare.net/SteveSfartz/building-microservices-55458071
A Guide on What Are Microservices: Pros, Cons, Use Cases, and MoreSimform
IT organizations can be benefitted from a microservices approach to application development with more agile and accelerated time to market. However, there is a catch in order to break an app into fine-grained services.
Join our webinar to learn how BMC enables enterprise customers to transform their organization to a digital DevOps model. In this webinar, NICE InContact will share their story of transforming from a physical data center to a hybrid environment, including data center and cloud infrastructure. They'll talk about strategies, challenges, best practices, and navigating an acquisition. Learn how NICE InContact improved service management and capacity management as well as resolution time for incidents using BMC TrueSight and AWS DevOps solutions. Join us to learn from a peer in the enterprise space to help you plan your own IT infrastructure transformation.
Similar to Redis and Kafka - Advanced Microservices Design Patterns Simplified (20)
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Generative AI Deep Dive: Advancing from Proof of Concept to Production
Redis and Kafka - Advanced Microservices Design Patterns Simplified
1. Redis and Kafka
Advanced Microservices Design Patterns Simplified
ALLEN TERLETO
SENIOR DIRECTOR, CUSTOMER SOLUTIONS AND FIELD ENGINEERING
https://www.linkedin.com/in/allenterleto/
AUGUST 2020
2. 2
Explore demand and goals for Microservices
1
Microservices
Architecture
Define the characteristics of a Microservice
2
Quick introduction of Kafka and Redis
3
Advanced Microservices Architecture Patterns
4
+
3.
4. What is holding back organizations from improving time-to-market?
4
Some Manager: We need more communication on and between our teams
Jeff Bezos: “No. Communication is terrible”
5. Software Evolution from Monolithic to Microservices
5 Source: https://www.martinfowler.com/microservices/
6. Characteristics of Microservices Architecture
6
Source: https://www.martinfowler.com/microservices/
1. Componentization via Services
2. Organized around Business Capabilities
3. Smart endpoints & dumb pipes
4. Decentralized Governance
5. Decentralized Data Management
6. Products not Projects
7. Infrastructure Automation
8. Design for failure
9. Evolutionary Design
API Gateway
Microservice
Message Broker
Microservice Microservice
9. Open Source In-Memory Database
supporting a variety of data structures and
data models optimized for
real-time and high-throughput use-cases
Open Source Stream Processing Platform
supporting distributed messaging
publish-subscribe data feeds for near-
real-time and high-throughput use-cases
9
10. Open Source Redis Modules
10 For more info check out: https://redislabs.com/community/redis-modules-hub/
12. Design Pattern - Bounded Context | Domain Driven Design
13
RedisAIRediSearch
risk:profile
input:tensors
output:tensors
RedisBloom
Authenticate
Digital Identity
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
Probabilistic Fraud
Detection Checkpoint
Transaction
Risk Scoring
user:transcations
Approve | Decline
Payment
Solution
• Practice Domain-Driven Design (DDD) principles
• Simplify complex domain models by dividing them into
separate Bounded Contexts; manageable by small teams
• Choose optimal data model based on each microservice’s data
access patterns and SLAs (RediSearch, RedisBloom, RedisAI, etc.)
Challenge
• Efficient communication between software
developers and domain experts
• Smaller autonomous teams are more productive
• Common terminology to establish requirements,
acceptance criteria, test cases, etc.
Bounded Context Bounded Context
13. Design Anti-Pattern – Two-Phase Commit Transactions
15
REDIS STREAMS
API Gateway
Microservice Microservice
write write
Challenge
• Each microservice can have its own database
however some business transactions will span
multiple Business Contexts
• In a Microservices Architecture, the Two-Phase
Commit protocol is an anti-pattern, since one or
more out of the hundreds or thousands of
microservices can, at any time, become unavailable
commit commitSolution
• Embrace eventual consistency and change mental model
from conforming to traditional ACID constraints
• Apply appropriate design patterns –
Message Broker, Saga, CQRS, Message Relay, Outbox, etc.
14. Design Pattern – Publish-Subscribe Message Broker
16
Kafka Message Broker
RediSearch RedisBloom
API Gateway
Authenticate
Digital Identity
Probabilistic Fraud
Detection Checkpoint
Challenge
• Two-Phase Commit protocol is an anti-pattern
• Transactions will span multiple business contexts
• Microservices can become unavailable at anytime
Solution
• Embrace event-driven architecture principles
• Inter-service communication (events) asynchronously
flow through a publish-subscribe Message Broker
• Zero, one, or more microservices can subscribe to
events that are relevant to their Business Context;
each at their own pace
• Microservices, recently recovered from unavailability, can
catch up to all the events they missed while unavailable
publish
subscribe subscribe
15. Design Pattern – Choreography-based Saga (State Machine)
17
Kafka Message Broker
RedisAIRedisBloom
Probabilistic Fraud
Detection Checkpoint
Transaction
Risk Scoring
Approve | Decline
Payment
YesNo
YesNo
Challenge
• Transactions will span multiple Business Contexts and the Two-Phase Commit protocol is an anti-pattern
• Microservices can include decision-logic that impacts which downstream services are in the transactional scope
Solution
• Saga is a sequence of local transactions
• Choreography-based Sagas are not
centrally coordinated
• Has roots in Finite-State Machine
• Each local transaction updates its
database and publishes an event to
trigger the next local transaction
• Sagas allow for unavailability along
their Chain-of-Responsibility, rollback
compensation for failures, and can replay the entire transaction
16. Design Pattern – Transactional Outbox | Message Relay
18
RediSearch
Kafka Message Broker
Authenticate
Digital Identity
RedisGears
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
Challenge
• Writing atomically to a database and a message broker can add
unnecessary code complexity
• Message Broker unavailability would force decisions about each
microservice’s trade-offs between availability and consistency
Solution
• Separate the concern of managing inter-service communication
by using an event-driven Message Relay
• RedisGears can be used to easily implement the Transactional
Outbox pattern by listening for events within Redis databases and
relaying event notifications (write-behind pattern) to a Message
Broker for downstream inter-service communication
• Redis-based microservices would avoid code complexity, concerns
with Message Broker unavailability, and performance overhead
17. Design Pattern – Capturing Telemetry
19
RedisGears
RedisTimeSeries
Analytics Dashboard
RediSearch
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
RedisAI
risk:profile
input:tensors
output:tensors
RedisBloom
user:transcations
RedisGears RedisGears
RedisTimeSeries
Challenge
• Telemetry is built on three pillars –
metrics, logs, and tracing
• Metrics are used for holistic observability
of a distributed-system in real time
• Metrics and tracing visualization often use
an underlying data model called “time-series”
Solution
• RedisTimeSeries has a native time-series data
model supporting high-volume inserts, low-
latency reads, aggregated queries, search-by
labels, downsampling, retention, indexing, etc.
• Metrics can be collected at various levels –
KPIs, application SLAs, infrastructure utilization, etc.
18. How can developers visualize time-series telemetry data?
20 Download for free: https://redislabs.com/redisinsight/
19. How can business analysts visualize time-series telemetry data?
21 For more info check out: https://github.com/mikhailredis/redis-pop-up-store and https://github.com/RedisTimeSeries/grafana-redis-datasource
20. Design Pattern – Event Sourcing
22
RedisAI
Analytics Dashboard
Redis Streams
risk:profile
input:tensors
output:tensors
RedisBloom
user:transcations
RedisGears RedisGears
Kafka Message Broker
Challenge
• Observability, tracing, and auditing of Choreography-based
Sagas is difficult across a decoupled chain of microservices
Solution
• Event Sourcing implies recording every microservice’s state
change as an immutable event; acting as a source of truth
• It is composed of a Message Broker and Event Store
• Based on the ordered sequence of events persisted in the
event store, a microservice can replay/rebuild its system
state by reprocessing recorded events at any time
• Redis Streams is an immutable in-memory append-only log
data structure perfectly suited and used as an Event Store
• Visualizing a stream of events can provide real-time
observability of the system’s state and instant recovery/replay
22. Design Pattern – CQRS (Command Query Responsibility Segregation)
24
API Gateway
Approve | Decline
Payment
Payment History
RedisCDC
Redis Enterprise
readwrite
Challenge
• Each microservice should choose an optimal data model
for its unique data access patterns and SLAs
• There is no single data model optimally suited for all needs
Solution
• CQRS implies maintaining separate data structures for
writing (command) and reading (query) information
• Allows a microservice with extreme durability SLAs to use a
write-optimized database, while another microservice that
requires access to the same data uses a read-optimized database
• Managing eventual consistency between models is commonly
handled using the Transaction Log Tailing pattern
(a.k.a. Change-Data-Capture) or Message Relay pattern (covered earlier)
• RedisCDC provides a seamless drop-in-solution to keep heterogenous databases in-synch with Redis
23. Design Pattern – Shared Data (Microservice Level)
25
API Gateway
Approve | Decline
Payment
Payment History
RedisCDC
Redis Enterprise
readwrite
Clearing & Settlement
read
Challenge
• Multiple microservices need to
query the same data which
could be sourced from multiple
downstream microservices
Solution
• Implement the CQRS pattern
to build a view-only database
composed of replicated state-
change events from one or more
source databases
• Redis is optimally suited as a view-only
database, or microservices-cache,
because it inherently guarantees atomicity
for each operation, across multiple clients,
and provides sub-millisecond latency at scale
24. Design Pattern – Shared Data (Global Level)
26
Redis Enterprise
user:session
oauth:token
Kafka Message Broker
RediSearch RedisBloom
API Gateway
Authenticate
Digital Identity
Probabilistic Fraud
Detection Checkpoint
Challenge
• Session state, client authentication tokens,
and/or other global data may need to
be shared across hundreds of microservices
• A shared database, or ephemeral cache,
would couple all microservices together
• A global database could become
a single-point of failure user:transcations
user:profile
behavioral:profile
transaction:profile
geolocation
trusted:devices
Solution
• API Gateway deploys its own isolated and
highly-available database to store global data
• Redis is used across thousands of
microservices architectures in production
to manage session state and client authentication tokens
25. Retrospective
27
Challenges
• There’s more to microservices than breaking off chunks of code from a monolith
• Data Management quickly becomes a pain point if best practices are not considered
• Decentralization comes with trades-offs to complexity, technology sprawl, observability, etc.
Solutions
• Microservices design patterns help avoid complexity at scale
• Redis supports multiple models optimally suitable to simplify their implementation
• Kafka and Redis help turn Microservices Architecture’s beautiful chaos into controlled chaos
26. Check out my interview with Chris Richardson from
28
https://www.youtube.com/watch?v=q5Z8-cwGBXQ
27. Free Redis Microservices e-book for Developers
29 https://redislabs.com/docs/redis-microservices-for-dummies/
28. Check out these free resources to learn more about Redis
30
• http://redis.io/
• https://university.redislabs.com/
• https://docs.redislabs.com/latest/rs/
• https://www.youtube.com/c/RedisUniversity