Speed-Up Kafka Delivery with AsyncAPI & Microcks | Hugo Guerrero, Red Hat
•
0 likes•979 views
Apache Kafka is getting used as an event backbone in new organizations everyday. We would love to send every byte of data through the event bus. Like traditional REST APIs a contract-first approach is very useful when designing event-driven architectures. In the case of asynchronous APIs, we have the AsynAPI specification to document the endpoints where the schema of the records become the main part of the contract payload. Microcks allows us to deploy a testing and mocking platform to have a unified view of the endpoints to speed-up application delivery.
Report
Share
Report
Share
Download to read offline
Recommended
Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | F...
With Apache Kafka’s rise for event-driven architectures, developers require a specification to design effective event-driven APIs. AsyncAPI has been developed based on OpenAPI to define the endpoints and schemas of brokers and topics. For Kafka applications, the broker’s design to handle high throughput serialized payloads brings challenges for consumers and producers managing the structure of the message. For this reason, a registry becomes critical to achieve schema governance. Apicurio Registry is an end-to-end solution to store API definitions and schemas for Kafka applications. The project includes serializers, deserializers, and additional tooling. The registry supports several types of artifacts including OpenAPI, AsyncAPI, GraphQL, Apache Avro, Google protocol buffers, JSON Schema, Kafka Connect schema, WSDL, and XML Schema (XSD). It also checks them for validity and compatibility.
In this session, we will be covering the following topics:
● The importance of having a contract-first approach to event-driven APIs
● What is AsyncAPI, and how it helps to define Kafka endpoints and schemas
● The Kafka challenges on message structure when serializing and deserializing
● Introduction to Apicurio Registry and schema management for Kafka
● Examples of how to use Apicurio Registry with popular Java frameworks like Spring and Quarkus
What is an API Gateway?
Irfan Baqui, Senior Engineer at LunchBadger, breaks down the important role of the API Gateway in Microservices. Additionally, Irfan covers how to get started with Express Gateway, an open source API Gateway built entirely on Express.js. Originally presented at the San Francisco Node Meetup.
Async API and Solace: Enabling the Event-Driven Future
Fran Méndez, founder of AsyncAPI, and Jonathan Schabowsky, senior architect at Solace, explain how the two companies are working together in this presentation from Gartner AADI.
The Architecture of an API Platform
My presentation from Nordic APIs 2014 in Stockholm, Sweden.
How can the architecture of one API platform look like? How can you break down things to make this challenge easier?
Agile, User Stories, Domain Driven Design
Building Cloud-Native App Series - Part 1 of 11
Microservices Architecture Series
Design Thinking, Lean Startup, Agile (Kanban, Scrum),
User Stories, Domain-Driven Design
Grafana optimization for Prometheus
This document discusses using Grafana to optimize visualization of metrics from Prometheus in a dynamic environment. It describes deploying multiple Prometheus instances to monitor over 100 instances per service across various services running on EC2. Key Grafana features discussed include templating to dynamically filter dashboards, panel repetition to show multiple graphs, and scripted dashboards to generate dashboards from JSON definitions. The document provides examples of using these features to create service trend dashboards, dynamically refresh dashboards based on time range changes, switch data sources, and generate alert dashboards from Prometheus alert views.
Event Sourcing & CQRS, Kafka, Rabbit MQ
Building Cloud-Native App Series - Part 2 of 11
Microservices Architecture Series
Event Sourcing & CQRS,
Kafka, Rabbit MQ
Case Studies (E-Commerce App, Movie Streaming, Ticket Booking, Restaurant, Hospital Management)
Api Gateway
API Gateways are the well suited service for microservices architecture. It provides many security and performance related features along with reliability of the system. These slides explains what is API Gateway. What is microservices architecture, its benefits and how API Gateway empower this architecture. Further more API Gateway aggregation is explained with an example.
Recommended
Apicurio Registry: Event-driven APIs & Schema governance for Apache Kafka | F...
With Apache Kafka’s rise for event-driven architectures, developers require a specification to design effective event-driven APIs. AsyncAPI has been developed based on OpenAPI to define the endpoints and schemas of brokers and topics. For Kafka applications, the broker’s design to handle high throughput serialized payloads brings challenges for consumers and producers managing the structure of the message. For this reason, a registry becomes critical to achieve schema governance. Apicurio Registry is an end-to-end solution to store API definitions and schemas for Kafka applications. The project includes serializers, deserializers, and additional tooling. The registry supports several types of artifacts including OpenAPI, AsyncAPI, GraphQL, Apache Avro, Google protocol buffers, JSON Schema, Kafka Connect schema, WSDL, and XML Schema (XSD). It also checks them for validity and compatibility.
In this session, we will be covering the following topics:
● The importance of having a contract-first approach to event-driven APIs
● What is AsyncAPI, and how it helps to define Kafka endpoints and schemas
● The Kafka challenges on message structure when serializing and deserializing
● Introduction to Apicurio Registry and schema management for Kafka
● Examples of how to use Apicurio Registry with popular Java frameworks like Spring and Quarkus
What is an API Gateway?
Irfan Baqui, Senior Engineer at LunchBadger, breaks down the important role of the API Gateway in Microservices. Additionally, Irfan covers how to get started with Express Gateway, an open source API Gateway built entirely on Express.js. Originally presented at the San Francisco Node Meetup.
Async API and Solace: Enabling the Event-Driven Future
Fran Méndez, founder of AsyncAPI, and Jonathan Schabowsky, senior architect at Solace, explain how the two companies are working together in this presentation from Gartner AADI.
The Architecture of an API Platform
My presentation from Nordic APIs 2014 in Stockholm, Sweden.
How can the architecture of one API platform look like? How can you break down things to make this challenge easier?
Agile, User Stories, Domain Driven Design
Building Cloud-Native App Series - Part 1 of 11
Microservices Architecture Series
Design Thinking, Lean Startup, Agile (Kanban, Scrum),
User Stories, Domain-Driven Design
Grafana optimization for Prometheus
This document discusses using Grafana to optimize visualization of metrics from Prometheus in a dynamic environment. It describes deploying multiple Prometheus instances to monitor over 100 instances per service across various services running on EC2. Key Grafana features discussed include templating to dynamically filter dashboards, panel repetition to show multiple graphs, and scripted dashboards to generate dashboards from JSON definitions. The document provides examples of using these features to create service trend dashboards, dynamically refresh dashboards based on time range changes, switch data sources, and generate alert dashboards from Prometheus alert views.
Event Sourcing & CQRS, Kafka, Rabbit MQ
Building Cloud-Native App Series - Part 2 of 11
Microservices Architecture Series
Event Sourcing & CQRS,
Kafka, Rabbit MQ
Case Studies (E-Commerce App, Movie Streaming, Ticket Booking, Restaurant, Hospital Management)
Api Gateway
API Gateways are the well suited service for microservices architecture. It provides many security and performance related features along with reliability of the system. These slides explains what is API Gateway. What is microservices architecture, its benefits and how API Gateway empower this architecture. Further more API Gateway aggregation is explained with an example.
Building Event Driven (Micro)services with Apache Kafka
What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so.
Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Spring Boot+Kafka: the New Enterprise Platform
This document discusses how Spring Boot and Kafka can form the basis of a new enterprise application platform focused on continuous delivery, event-driven architectures, and streaming data. It provides examples of companies that have successfully adopted this approach, such as Netflix transitioning to Spring Boot and a banking brand building a new core banking system using Spring Streams and Kafka. The document advocates an "event-first" and microservices-oriented mindset enabled by a streaming data platform and suggests that Spring Boot, Kafka, and related technologies provide a turnkey solution for implementing this new application development approach at large enterprises.
The Top 5 Apache Kafka Use Cases and Architectures in 2022
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
Building Event-Driven (Micro) Services with Apache Kafka
The document discusses building event-driven microservices using Apache Kafka. It describes how microservices can interact asynchronously through events published to an event hub like Apache Kafka. This allows for loose coupling between services and the ability to integrate new services by consuming past events from the event log. The document also discusses how Apache Kafka can be used for change data capture from legacy systems, streaming data integration, and providing a unified platform for real-time event processing and historical analytics.
API Maturity Model (Webcast with Accenture)
The document describes Accenture's API Maturity Model, which provides a framework to help organizations develop and manage their APIs from an initial "ad hoc" stage to a fully "industrialized" stage. The model outlines five stages of maturity - ad hoc, organize, tactical, mission critical, and industrial. For each stage, it describes key characteristics and capabilities an organization should develop in areas like strategy, architecture, development process, community management, and optimization. The goal of the model is to help organizations assess their current API maturity and identify steps to progress along the maturity curve to better enable and manage their APIs and digital ecosystems.
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments
Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. This session gives an overview of several scenarios that may require multi-cluster solutions and discusses real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka.
Key takeaways:
In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments.
Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs.
Learn about features and limitations of Kafka for multi cluster deployments
Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception.
02 api gateway
This document discusses API gateways as a solution for challenges that arise in microservices architectures. It describes how a monolithic architecture can become complex as services grow quickly. In a microservices architecture, clients could communicate directly with each service but this introduces problems around endpoint management, multiple requests, and refactoring difficulties. An API gateway provides a single entry point, routes requests to appropriate services, and aggregates results to address these issues. It then demonstrates Netflix Zuul, an open source API gateway, and provides a demo of its use with Eureka service discovery and routing between hello and goodbye microservices.
Microservices Architecture & Testing Strategies
The document discusses Microservices architecture and compares it to monolithic architecture. It covers topics like infrastructure for Microservices including API gateways, service discovery, event buses. It also discusses design principles like domain-driven design, event sourcing and CQRS. Microservices are presented as a better approach as they allow independent deployments, scale independently and use multiple programming languages compared to monolithic applications.
Microservices Testing Strategies JUnit Cucumber Mockito Pact
Building Cloud-Native App Series - Part 6 of 11
Microservices Architecture Series
Testing Strategies
- JUnit 5
- Cucumber BDD
- Selenium
- SpringBoot Test
- Mockito
- Wiremock
- Pact
Microservice Architecture
The document discusses microservice architecture, including concepts, benefits, principles, and challenges. Microservices are an architectural style that structures an application as a collection of small, independent services that communicate with each other, often using RESTful API's. The approach aims to overcome limitations of monolithic architectures like scalability and allow for independent deployments. The key principles include organizing services around business domains, automating processes, and designing services to be independently deployable.
Producer Performance Tuning for Apache Kafka
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with inflexible and non-scalable technologies like REST / HTTP. This video takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.
Key takeaways:
- Apache Kafka decouples services, including event streams and request-response
- Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
- Service Mesh helps with security and observability at ecosystem / organization scale
- Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
Blog post: http://www.kai-waehner.de/blog/2019/09/24/cloud-native-apache-kafka-kubernetes-envoy-istio-linkerd-service-mesh
Video recording of this slide deck: https://youtu.be/Us_C4RFOUrA
이벤트 기반 분산 시스템을 향한 여정
배민찬(https://www.baeminchan.com) 서비스의 백엔드 시스템 중 일부가 지난 1년간 어떤 고민과 아이디어, 결과물을 만들어냈는지 공유하려고 합니다. 발표 중 언급되는 용어나 도구에 대해 일반적인 정의나 간단한 설명은 언급되나 자세히 다루지 않습니다. 사용된 도구들로 어떻게 이벤트 기반 분산 시스템을 만들었는지에 대한 이야기가 중심입니다.
Understanding MicroSERVICE Architecture with Java & Spring Boot
This is a deep journey into the realm of "microservice architecture", and in that I will try to cover each inch of it, but with a fixed tech stack of Java with Spring Cloud. Hence in the end, you will be get know each and every aspect of this distributed design, and will develop an understanding of each and every concern regarding distributed system construct.
Mule ESB - Integration Simplified
This presentation discusses Mule ESB and how to simplify integration. It briefly mentions a brief history of integration, information silos, SOA. It also highlights several integration patterns.
Cast vs sonar
This document compares the software quality analysis tools CAST and SONAR. It finds that CAST covers more functionality overall, covering 80% of all functionality compared to 60% for SONAR. However, SONAR has some advantages in testing capabilities. Both tools cover the main technologies used at Amadeus, like Java and C++, but CAST supports more technologies. While CAST has more features, it also has higher license costs compared to the open source SONAR.
Multi cluster, multitenant and hierarchical kafka messaging service slideshare
Multi cluster, multitenant and hierarchical kafka messaging service slideshareAllen (Xiaozhong) Wang
This document discusses Netflix's approach to scaling Apache Kafka through a hierarchical, multi-cluster architecture. It addresses the challenges of scaling a single Kafka cluster, such as increased latency and vulnerability to failures. Netflix uses multiple "fronting" Kafka clusters for data collection and buffering optimized for producers. Consumer clusters scale by adding brokers or moving partitions to new clusters. Producers and consumers can operate across multiple clusters simultaneously using routing services. This multi-cluster approach allows independent scaling of producers and consumers while balancing data distribution.Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...
With the increasing complexity that comes with adopting distributed architectures like Kafka, it is difficult to identify and triage performance anomalies. Implementing Distributed Tracing technologies like OpenTelemetry can add context to the messages flowing through your system, allowing us to visualize the data flow to pinpoint bottlenecks, bugs, and other performance issues. However, implementing end-to-end distributed tracing can be challenging for multi-request and asynchronous processes built on Kafka.
This talk will walk through how to use OpenTelemetry to tell the full story of a request as it travels through your Kafka producer, queue, and consumer. First, we will learn how context propagation works in OpenTelemetry with W3C and B3 protocols. Then, we will go through the problems faced when instrumenting asynchronous requests. Finally, we will go through the context propagation implementation of the OpenTelemetry Kafka exporter, tracing multi-request and asynchronous processes.
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
This document discusses approaching API security for multicloud environments using an abstraction called "Metacloud." It notes that as cloud deployments become more complex with multiple APIs, security risks increase. The document proposes addressing this by abstracting resources across clouds to reduce complexity and enable common security practices. This involves automating API access, data processing, services, and platforms to create a unified "Metacloud" or "Supercloud." The goal is to orchestrate security, observability, access management and other functions to help manage risk at scale across multiple cloud providers.
AWS ECS vs EKS
The document discusses container orchestration tools on AWS including Amazon ECS and Amazon EKS. It provides an overview of ECS, describing it as a fully managed container orchestration platform. It also summarizes EKS, noting that it is fully managed Kubernetes that allows users to run Kubernetes on AWS. The document compares ECS and EKS, noting differences in cost, ease of use, support, and compatibility between the two services.
Automated Apache Kafka Mocking and Testing with AsyncAPI | Hugo Guerrero, Red...
Apache Kafka is getting used as an event backbone in new organizations every day. We would love to send every byte of data through the event bus. Like traditional REST APIs, a contract-first approach is very useful when designing event-driven architectures. In the case of asynchronous APIs, we have the AsyncAPI specification to document the endpoints where the schema of the records become the main part of the contract payload. Microcks allows us to deploy a testing and mocking platform to have a unified view of the endpoints to speed-up application delivery.
In this session we will:
- Go over the evolution of API specifications
- Review the approach for contract-first design with Apache Kafka
- Introduce the AsyncAPI specification
- Take an overview of an implementation example for automated mocking and testing
Apidays New York 2024 - Increase your productivity with no-code GraphQL mocki...
Increase your productivity with no-code GraphQL mocking
Hugo Guerrero, Chief Software Architect, APIs & Integration Developer Advocate at Red Hat
Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024)
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
More Related Content
What's hot
Building Event Driven (Micro)services with Apache Kafka
What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will start with quick recap of how we created systems over the past 20 years and how different architectures evolved from it. The talk will show how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so.
Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Spring Boot+Kafka: the New Enterprise Platform
This document discusses how Spring Boot and Kafka can form the basis of a new enterprise application platform focused on continuous delivery, event-driven architectures, and streaming data. It provides examples of companies that have successfully adopted this approach, such as Netflix transitioning to Spring Boot and a banking brand building a new core banking system using Spring Streams and Kafka. The document advocates an "event-first" and microservices-oriented mindset enabled by a streaming data platform and suggests that Spring Boot, Kafka, and related technologies provide a turnkey solution for implementing this new application development approach at large enterprises.
The Top 5 Apache Kafka Use Cases and Architectures in 2022
This document discusses the top 5 use cases and architectures for data in motion in 2022. It describes:
1) The Kappa architecture as an alternative to the Lambda architecture that uses a single stream to handle both real-time and batch data.
2) Hyper-personalized omnichannel experiences that integrate customer data from multiple sources in real-time to provide personalized experiences across channels.
3) Multi-cloud deployments using Apache Kafka and data mesh architectures to share data across different cloud platforms.
4) Edge analytics that deploy stream processing and Kafka brokers at the edge to enable low-latency use cases and offline functionality.
5) Real-time cybersecurity applications that use streaming data
Building Event-Driven (Micro) Services with Apache Kafka
The document discusses building event-driven microservices using Apache Kafka. It describes how microservices can interact asynchronously through events published to an event hub like Apache Kafka. This allows for loose coupling between services and the ability to integrate new services by consuming past events from the event log. The document also discusses how Apache Kafka can be used for change data capture from legacy systems, streaming data integration, and providing a unified platform for real-time event processing and historical analytics.
API Maturity Model (Webcast with Accenture)
The document describes Accenture's API Maturity Model, which provides a framework to help organizations develop and manage their APIs from an initial "ad hoc" stage to a fully "industrialized" stage. The model outlines five stages of maturity - ad hoc, organize, tactical, mission critical, and industrial. For each stage, it describes key characteristics and capabilities an organization should develop in areas like strategy, architecture, development process, community management, and optimization. The goal of the model is to help organizations assess their current API maturity and identify steps to progress along the maturity curve to better enable and manage their APIs and digital ecosystems.
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka deployments
Multi-cluster and cross-data center deployments of Apache Kafka have become the norm rather than an exception. This session gives an overview of several scenarios that may require multi-cluster solutions and discusses real-world examples with their specific requirements and trade-offs, including disaster recovery, aggregation for analytics, cloud migration, mission-critical stretched deployments and global Kafka.
Key takeaways:
In many scenarios, one Kafka cluster is not enough. Understand different architectures and alternatives for multi-cluster deployments.
Zero data loss and high availability are two key requirements. Understand how to realize this, including trade-offs.
Learn about features and limitations of Kafka for multi cluster deployments
Global Kafka and mission-critical multi-cluster deployments with zero data loss and high availability became the normal, not an exception.
02 api gateway
This document discusses API gateways as a solution for challenges that arise in microservices architectures. It describes how a monolithic architecture can become complex as services grow quickly. In a microservices architecture, clients could communicate directly with each service but this introduces problems around endpoint management, multiple requests, and refactoring difficulties. An API gateway provides a single entry point, routes requests to appropriate services, and aggregates results to address these issues. It then demonstrates Netflix Zuul, an open source API gateway, and provides a demo of its use with Eureka service discovery and routing between hello and goodbye microservices.
Microservices Architecture & Testing Strategies
The document discusses Microservices architecture and compares it to monolithic architecture. It covers topics like infrastructure for Microservices including API gateways, service discovery, event buses. It also discusses design principles like domain-driven design, event sourcing and CQRS. Microservices are presented as a better approach as they allow independent deployments, scale independently and use multiple programming languages compared to monolithic applications.
Microservices Testing Strategies JUnit Cucumber Mockito Pact
Building Cloud-Native App Series - Part 6 of 11
Microservices Architecture Series
Testing Strategies
- JUnit 5
- Cucumber BDD
- Selenium
- SpringBoot Test
- Mockito
- Wiremock
- Pact
Microservice Architecture
The document discusses microservice architecture, including concepts, benefits, principles, and challenges. Microservices are an architectural style that structures an application as a collection of small, independent services that communicate with each other, often using RESTful API's. The approach aims to overcome limitations of monolithic architectures like scalability and allow for independent deployments. The key principles include organizing services around business domains, automating processes, and designing services to be independently deployable.
Producer Performance Tuning for Apache Kafka
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Microservice architectures are not free lunch! Microservices need to be decoupled, flexible, operationally transparent, data aware and elastic. Most material from last years only discusses point-to-point architectures with inflexible and non-scalable technologies like REST / HTTP. This video takes a look at cutting edge technologies like Apache Kafka, Kubernetes, Envoy, Linkerd and Istio to implement a cloud-native service mesh to solve these challenges and bring microservices to the next level of scale, speed and efficiency.
Key takeaways:
- Apache Kafka decouples services, including event streams and request-response
- Kubernetes provides a cloud-native infrastructure for the Kafka ecosystem
- Service Mesh helps with security and observability at ecosystem / organization scale
- Envoy and Istio sit in the layer above Kafka and are orthogonal to the goals Kafka addresses
Blog post: http://www.kai-waehner.de/blog/2019/09/24/cloud-native-apache-kafka-kubernetes-envoy-istio-linkerd-service-mesh
Video recording of this slide deck: https://youtu.be/Us_C4RFOUrA
이벤트 기반 분산 시스템을 향한 여정
배민찬(https://www.baeminchan.com) 서비스의 백엔드 시스템 중 일부가 지난 1년간 어떤 고민과 아이디어, 결과물을 만들어냈는지 공유하려고 합니다. 발표 중 언급되는 용어나 도구에 대해 일반적인 정의나 간단한 설명은 언급되나 자세히 다루지 않습니다. 사용된 도구들로 어떻게 이벤트 기반 분산 시스템을 만들었는지에 대한 이야기가 중심입니다.
Understanding MicroSERVICE Architecture with Java & Spring Boot
This is a deep journey into the realm of "microservice architecture", and in that I will try to cover each inch of it, but with a fixed tech stack of Java with Spring Cloud. Hence in the end, you will be get know each and every aspect of this distributed design, and will develop an understanding of each and every concern regarding distributed system construct.
Mule ESB - Integration Simplified
This presentation discusses Mule ESB and how to simplify integration. It briefly mentions a brief history of integration, information silos, SOA. It also highlights several integration patterns.
Cast vs sonar
This document compares the software quality analysis tools CAST and SONAR. It finds that CAST covers more functionality overall, covering 80% of all functionality compared to 60% for SONAR. However, SONAR has some advantages in testing capabilities. Both tools cover the main technologies used at Amadeus, like Java and C++, but CAST supports more technologies. While CAST has more features, it also has higher license costs compared to the open source SONAR.
Multi cluster, multitenant and hierarchical kafka messaging service slideshare
Multi cluster, multitenant and hierarchical kafka messaging service slideshareAllen (Xiaozhong) Wang
This document discusses Netflix's approach to scaling Apache Kafka through a hierarchical, multi-cluster architecture. It addresses the challenges of scaling a single Kafka cluster, such as increased latency and vulnerability to failures. Netflix uses multiple "fronting" Kafka clusters for data collection and buffering optimized for producers. Consumer clusters scale by adding brokers or moving partitions to new clusters. Producers and consumers can operate across multiple clusters simultaneously using routing services. This multi-cluster approach allows independent scaling of producers and consumers while balancing data distribution.Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...
With the increasing complexity that comes with adopting distributed architectures like Kafka, it is difficult to identify and triage performance anomalies. Implementing Distributed Tracing technologies like OpenTelemetry can add context to the messages flowing through your system, allowing us to visualize the data flow to pinpoint bottlenecks, bugs, and other performance issues. However, implementing end-to-end distributed tracing can be challenging for multi-request and asynchronous processes built on Kafka.
This talk will walk through how to use OpenTelemetry to tell the full story of a request as it travels through your Kafka producer, queue, and consumer. First, we will learn how context propagation works in OpenTelemetry with W3C and B3 protocols. Then, we will go through the problems faced when instrumenting asynchronous requests. Finally, we will go through the context propagation implementation of the OpenTelemetry Kafka exporter, tracing multi-request and asynchronous processes.
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
This document discusses approaching API security for multicloud environments using an abstraction called "Metacloud." It notes that as cloud deployments become more complex with multiple APIs, security risks increase. The document proposes addressing this by abstracting resources across clouds to reduce complexity and enable common security practices. This involves automating API access, data processing, services, and platforms to create a unified "Metacloud" or "Supercloud." The goal is to orchestrate security, observability, access management and other functions to help manage risk at scale across multiple cloud providers.
AWS ECS vs EKS
The document discusses container orchestration tools on AWS including Amazon ECS and Amazon EKS. It provides an overview of ECS, describing it as a fully managed container orchestration platform. It also summarizes EKS, noting that it is fully managed Kubernetes that allows users to run Kubernetes on AWS. The document compares ECS and EKS, noting differences in cost, ease of use, support, and compatibility between the two services.
What's hot (20)
Building Event Driven (Micro)services with Apache Kafka
Building Event Driven (Micro)services with Apache Kafka
The Top 5 Apache Kafka Use Cases and Architectures in 2022
The Top 5 Apache Kafka Use Cases and Architectures in 2022
Building Event-Driven (Micro) Services with Apache Kafka
Building Event-Driven (Micro) Services with Apache Kafka
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Microservices Testing Strategies JUnit Cucumber Mockito Pact
Microservices Testing Strategies JUnit Cucumber Mockito Pact
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Service Mesh with Apache Kafka, Kubernetes, Envoy, Istio and Linkerd
Understanding MicroSERVICE Architecture with Java & Spring Boot
Understanding MicroSERVICE Architecture with Java & Spring Boot
Multi cluster, multitenant and hierarchical kafka messaging service slideshare
Multi cluster, multitenant and hierarchical kafka messaging service slideshare
Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...
Distributed Tracing for Kafka with OpenTelemetry with Daniel Kim | Kafka Summ...
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
APIsecure 2023 - Approaching Multicloud API Security USing Metacloud, David L...
Similar to Speed-Up Kafka Delivery with AsyncAPI & Microcks | Hugo Guerrero, Red Hat
Automated Apache Kafka Mocking and Testing with AsyncAPI | Hugo Guerrero, Red...
Apache Kafka is getting used as an event backbone in new organizations every day. We would love to send every byte of data through the event bus. Like traditional REST APIs, a contract-first approach is very useful when designing event-driven architectures. In the case of asynchronous APIs, we have the AsyncAPI specification to document the endpoints where the schema of the records become the main part of the contract payload. Microcks allows us to deploy a testing and mocking platform to have a unified view of the endpoints to speed-up application delivery.
In this session we will:
- Go over the evolution of API specifications
- Review the approach for contract-first design with Apache Kafka
- Introduce the AsyncAPI specification
- Take an overview of an implementation example for automated mocking and testing
Apidays New York 2024 - Increase your productivity with no-code GraphQL mocki...
Increase your productivity with no-code GraphQL mocking
Hugo Guerrero, Chief Software Architect, APIs & Integration Developer Advocate at Red Hat
Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024)
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Apidays Paris 2023 - Boosting Event-Driven Development with AsyncAPI and Micr...
Apidays Paris 2023 - Software and APIs for Smart, Sustainable and Sovereign Societies
December 6, 7 & 8, 2023
Boosting Event-Driven Development with AsyncAPI and Microcks
Hugo Guerrero, Chief Software Architect, APIs & Integration Developer Advocate at Red Hat
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays LIVE London 2021 - Getting started with Event-Driven APIs by Hugo Gue...
apidays LIVE London 2021 - Reaching Maximum Potential in Banking & Insurance with API Mindset
October 27 & 28, 2021
The future API stack : GraphQL, gRPC and API specifications
Getting started with Event-Driven APIs
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat
apidays LIVE Paris 2021 - Getting started with Event-Driven APis by Hugo Guer...
apidays LIVE Paris 2021 - APIs and the Future of Software
December 7, 8 & 9, 2021
Getting started with Event-Driven APis
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat
Building event-driven (Micro)Services with Apache Kafka
This talk begins with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dive into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
apidays LIVE Hong Kong 2021 - Event-driven APIs & Schema governance for Apach...
apidays LIVE Hong Kong 2021 - API Ecosystem & Data Interchange
August 25 & 26, 2021
Event-driven APIs & Schema governance for Apache Kafka
Hugo Guerrero, APIs & Messaging Developer Advocate at Red Hat
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Microservices with Kafka Ecosystem
Should you use traditional REST APIs to bind services together? Or is it better to use a richer, more loosely-coupled protocol? This talk will dig into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which. It highlights how the modern stream processing systems can be used to
hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Voxxed Days Microservices Paris 2018 - Keynote
In November 2010 Thomas Thwaites - a designer from the UK - gave a TED talk about his idea to build a toaster from scratch. He bought a cheap toaster from his local appliance store and was very surprised that this cheap toaster was consisting of 404 different parts made of 100+ materials. His journey was paved with failure and setbacks, but nine month later he came up with ... let's say with something that looked remotely like a toaster, but was only working for 2 seconds, before it melted down. Thomas Thwaites' journey reminded me in some kind of the struggles with transforming a monolith to microservices - especially in terms of the myriad of infrastructure complexities that comes with microservices. In this keynote I am talking about some options how a small team can handle infrastructure complexities and deliver business value.
This keynote was held at Voxxed Days Microservices in Paris on 29th October 2018.
C2 Matrix A Comparison of Command and Control Frameworks
Command and Control is one of the most important tactics in the MITRE ATT&CK matrix as it allows the attacker to interact with the target system and realize their objectives.
Organizations leverage Cyber Threat Intelligence to understand their threat model and adversaries that have the intent, opportunity, and capability to attack. Red Team, Blue Team, and virtual Purple Teams work together to understand the adversary Tactics, Techniques, and Procedures to perform adversary emulations and improve detective and preventive controls.
The C2 Matrix was created to aggregate all the Command and Control frameworks publicly available (open-source and commercial) in a single resource to assist teams in testing their own controls through adversary emulations (Red Team or Purple Team Exercises). Phase 1 lists all the Command and Control features such as the coding language used, channels (HTTP, TCP, DNS, SMB, etc.), agents, key exchange, and other operational security features and capabilities. This allows more efficient decisions making when called upon to emulate and adversary TTPs.
It is the golden age of Command and Control (C2) frameworks. Learn how these C2 frameworks work and start testing against your organization to improve detective and preventive controls.
The C2 Matrix currently has 35 command and control frameworks documented in a Google Sheet, web site, and questionnaire format.
Google Sheet (Golden Source):
https://docs.google.com/spreadsheets/d/1b4mUxa6cDQuTV2BPC6aA-GR4zGZi0ooPYtBe4IgPsSc/edit#gid=0
Website:
https://www.thec2matrix.com/matrix
https://ask.thec2matrix.com/
Follow on Twitter for updates:
https://twitter.com/c2_matrix
Serverless APIs with Apache OpenWhisk
The document discusses serverless computing and Apache OpenWhisk. It describes how OpenWhisk allows developers to focus on business logic rather than infrastructure by executing code in response to events in a serverless manner. OpenWhisk provides a programming model where developers can create actions to handle triggers via rules. A number of demos are presented showing how to create triggers, actions and rules with OpenWhisk to handle events and build REST APIs.
Apache Beam (incubating)
Presenter: Kenn Knowles, Software Engineer, Google & Apache Beam (incubating) PPMC member
Apache Beam (incubating) is a programming model and library for unified batch & streaming big data processing. This talk will cover the Beam programming model broadly, including its origin story and vision for the future. We will dig into how Beam separates concerns for authors of streaming data processing pipelines, isolating what you want to compute from where your data is distributed in time and when you want to produce output. Time permitting, we might dive deeper into what goes into building a Beam runner, for example atop Apache Apex.
Microservices with Kafka Ecosystem
This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each eachother in a Microservcies Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing systems can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Your Business, as APIs Leveraging Abstractions to Drive the Experience Economy
In this session, Asanka will explore how leveraging the correct level of abstractions can drive agile software manufacturing and improve customer experiences in the experience economy. By viewing their business as a set of APIs and using other abstraction layers, businesses can build modular and customer-centric systems that enable rapid innovation and growth. Attendees will gain insights into real-world examples of businesses leveraging APIs to drive growth and competitiveness. They will leave with practical strategies and tactics for streamlining software manufacturing processes and unlocking new revenue streams in the experience economy.
[apidays LIVE HONK KONG] - Building an Integrated Supply Chain for APIs
APIs are the products of the 21st century. As we build out API systems, we find that we are constantly learning from product journeys. We propose a new kind of supply chain - the Integrated Supply Chain for APIs (ISCA) - which is needed by any organization looking to create and monetize API products, either directly or indirectly. Through these slides, it is outlined our vision of the ISCA, identify five key patterns for success, and give a blueprint for creating a digital business based on API products.
Webinar - Big Data: Let's SMACK - Jorg Schad
The document discusses big data processing and the SMACK stack. It introduces Mesosphere and Apache Mesos as enabling distributed applications by multiplexing workloads across servers. It then covers components of the SMACK stack - including Apache Kafka for ingestion, Apache Spark for storage and analysis, Apache Cassandra for analytics, and Akka for acting on data. It discusses choosing messaging and stream processing systems and highlights Mesos support.
How to build a Distributed Serverless Polyglot Microservices IoT Platform us...
When people aren't talking about VMs and containers, they're talking about serverless architecture. Serverless is about no maintenance. It means you are not worried about low-level infrastructural and operational details. An event-driven serverless platform is a great use case for IoT.
In this session at @ThingsExpo, Animesh Singh, an STSM and Lead for IBM Cloud Platform and Infrastructure, detailed how to build a distributed serverless, polyglot, microservices framework using open source technologies like:
OpenWhisk: Open source distributed compute service to execute application logic in response to events
Docker: To run event driven actions 6. Ansible and BOSH: to deploy the serverless platform
MQTT: Messaging protocol for IoT
Node-RED: Tool to wire IoT together
Consul: Tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable.
Kafka: A high-throughput distributed messaging system.
StatsD/ELK/Graphite: For statistics, monitoring and logging
Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
The document summarizes Andrea Saltarello's presentation on implementing CQRS and event sourcing patterns on Azure. The presentation included a recap of CQRS and event sourcing, demonstrations of aggregates, handlers and read models, and discussions of deployment options on Azure including n-tiered and full-stack approaches. It also covered technology considerations and options for event buses, event stores and economic comparisons of Azure computing services.
Building Event-Driven (Micro)Services with Apache Kafka
Should we use traditional REST APIs to bind services together? Or is it better to use a more loosely-coupled protocol? This talk will dive into how we piece services together in event driven systems, how we use a distributed log (event hub) to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a traditional as well as in a stream processing fashion. The talk will show the difference between a request-driven and event-driven communication and show when to use which.
Similar to Speed-Up Kafka Delivery with AsyncAPI & Microcks | Hugo Guerrero, Red Hat (20)
Automated Apache Kafka Mocking and Testing with AsyncAPI | Hugo Guerrero, Red...
Automated Apache Kafka Mocking and Testing with AsyncAPI | Hugo Guerrero, Red...
Apidays New York 2024 - Increase your productivity with no-code GraphQL mocki...
Apidays New York 2024 - Increase your productivity with no-code GraphQL mocki...
Apidays Paris 2023 - Boosting Event-Driven Development with AsyncAPI and Micr...
Apidays Paris 2023 - Boosting Event-Driven Development with AsyncAPI and Micr...
apidays LIVE London 2021 - Getting started with Event-Driven APIs by Hugo Gue...
apidays LIVE London 2021 - Getting started with Event-Driven APIs by Hugo Gue...
apidays LIVE Paris 2021 - Getting started with Event-Driven APis by Hugo Guer...
apidays LIVE Paris 2021 - Getting started with Event-Driven APis by Hugo Guer...
Building event-driven (Micro)Services with Apache Kafka
Building event-driven (Micro)Services with Apache Kafka
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka Ecosystem
apidays LIVE Hong Kong 2021 - Event-driven APIs & Schema governance for Apach...
apidays LIVE Hong Kong 2021 - Event-driven APIs & Schema governance for Apach...
C2 Matrix A Comparison of Command and Control Frameworks
C2 Matrix A Comparison of Command and Control Frameworks
Your Business, as APIs Leveraging Abstractions to Drive the Experience Economy
Your Business, as APIs Leveraging Abstractions to Drive the Experience Economy
[apidays LIVE HONK KONG] - Building an Integrated Supply Chain for APIs
[apidays LIVE HONK KONG] - Building an Integrated Supply Chain for APIs
How to build a Distributed Serverless Polyglot Microservices IoT Platform us...
How to build a Distributed Serverless Polyglot Microservices IoT Platform us...
Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
Azure tales: a real world CQRS and ES Deep Dive - Andrea Saltarello
Building Event-Driven (Micro)Services with Apache Kafka
Building Event-Driven (Micro)Services with Apache Kafka
More from HostedbyConfluent
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way.
Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks.
Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case.
Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."
Renaming a Kafka Topic | Kafka Summit London
"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics.
Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"
Evolution of NRT Data Ingestion Pipeline at Trendyol
"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights.
We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string.
Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring.
The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service.
While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications.
The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection.
Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."
Exactly-once Stream Processing with Arroyo and Kafka
"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output.
Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing.
In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."
Fish Plays Pokemon | Kafka Summit London
"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium.
This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption.
By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."
Tiered Storage 101 | Kafla Summit London
What is tiered storage and what is it good for? After this session you will know how to leverage the tiered storage feature to enable longer retention than the storage attached to brokers allows. You will get acquainted with the different configuration options and know what to expect when you enable the feature, like for example when will the first upload to the remote object storage take place.
Building a Self-Service Stream Processing Portal: How And Why
"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work.
However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business.
You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want.
Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware.
What you’ll take away from our talk:
- What were the pain points in the previous environment
- How we transitioned to Confluent without service downtime
- Creating a self-service stream processing portal built on top of Connect and ksqlDB
- Use case of stream process portal"
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics.
Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application.
At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."
Navigating Private Network Connectivity Options for Kafka Clusters
"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option!
In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met.
I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines.
Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"
Explaining How Real-Time GenAI Works in a Noisy Pub
"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many.
LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data.
Large language models for your streaming application - explained with a little maths and a lot of pub stories"
TL;DR Kafka Metrics | Kafka Summit London
"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them.
This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."
A Window Into Your Kafka Streams Tasks | KSL
Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance.
- Introduction to Kafka producer internal and workflow
- Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance
- Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient.
- Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees.
- Q&A session with attendees to address specific questions and concerns."
Data Contracts Management: Schema Registry and Beyond
"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element.
In this talk, we’ll:
1. see why data contracts are so important but also difficult to implement;
2. identify the characteristics of a well-designed data contract:
discuss the anatomy of a data contract, its main elements and, how to formally describe them;
3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."
Code-First Approach: Crafting Efficient Flink Apps
"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications.
We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications.
Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."
Debezium vs. the World: An Overview of the CDC Ecosystem
"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you!
[1] https://debezium.io/"
Beyond Tiered Storage: Serverless Kafka with No Local Disks
"Separation of compute and storage has become the de-facto standard in the data industry for batch processing.
The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world.
In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.
Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it.
This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."
More from HostedbyConfluent (20)
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Recently uploaded
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more.
The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications.
We’ll discuss and demo the benefits of UiPath Apps and connectors including:
Creating a compelling user experience for any software, without the limitations of APIs.
Accelerating the app creation process, saving time and effort
Enjoying high-performance CRUD (create, read, update, delete) operations, for
seamless data management.
Speakers:
Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP
Charlie Greenberg, host
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Artificial Intelligence and Electronic Warfare
Artificial Intelligence and Electronic WarfarePapadakis K.-Cyber-Information Warfare Analyst & Cyber Defense/Security Consultant-Hellenic MoD
AI & Electronic WarfareFueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Leveraging the Graph for Clinical Trials and Standards
Katja Glaß
OpenStudyBuilder Community Manager - Katja Glaß Consulting
Marius Conjeaud
Principal Consultant - Neo4j
JavaLand 2024: Application Development Green Masterplan
My presentation slides I used at JavaLand 2024
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Recently uploaded (20)
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
JavaLand 2024: Application Development Green Masterplan
JavaLand 2024: Application Development Green Masterplan
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Speed-Up Kafka Delivery with AsyncAPI & Microcks | Hugo Guerrero, Red Hat
- 1. @MICROCKSIO #KAFKASUMMIT Speed-up Kafka delivery with AsyncAPI & Microcks Hugo Guerrero / @hguerreroo
- 2. @MICROCKSIO #KAFKASUMMIT Hugo Guerrero Mexican @ MA, USA @hguerreroo Event-Driven APIs Specialist Open Source Advocate History, Travel & Food Enthusiast
- 3. Events everywhere. Why the raise of events-driven API ? Photo by Carol Jeng on Unsplash #KAFKASUMMIT @MICROCKSIO
- 4. @MICROCKSIO #KAFKASUMMIT Because of distributed systems ! icons by Made x Made from the Noun Project TRADITIONAL REQUEST/RESPONSE API INVOLVE STRONG UPTIME COUPLING.
- 5. #KAFKASUMMIT @MICROCKSIO Kafkaesque Situation ENDPOINT DOCUMENTATION ? DATA FORMAT AND VALIDATION ? INFRASTRUCTURE ACCESS ? PUBLISHERS & CONSUMERS COMPLIANCE ? EVENTS AVAILABILITY ? MIS-UNDERSTANDING SLOW TIME TO MARKET, POOR QUALITY
- 6. #SOLACE SE icons by Made x Made from the Noun Project @MICROCKSIO HOW TO AVOID STRONG BUILD-TIME COUPLING ? Project Planning Architecture
- 7. @MICROCKSIO #KAFKASUMMIT Event-Driven Secret-Weapon An open source initiative that seeks to improve the current state of Event-Driven Architectures. Created to describe Event-driven APIs using MQTT, AMQP, Apache Kafka, ... “Sister” specification to OpenAPI: same base syntax, use JSON Schema or Avro underhood.
- 8. @MICROCKSIO #KAFKASUMMIT AsyncAPI Anatomy Info: metadata and documentation for this asynchronous API: version, title, description, contact and licensing terms Servers: list of brokers with binding details (URLs and security protocols) Channel Item: destination that will be used by the API to publish or consume events. Destination may be mapped to a topic, queue, address name depending on the protocol binding. Operation: whether the application using the AsyncAPI should subscribe or produce messages. Holds its own descriptive metadata Message: the content-type and structure of message headers and payload. May reference schemas of different types (AsyncAPI, JSON, Avro). May contain examples! Components: Common definitions (schemas, traits, messages) like in OpenAPI v3
- 9. @MICROCKSIO #KAFKASUMMIT Introducing Microcks Open source Kubernetes-native tool for API Mocking and Testing. It provides an enterprise-grade solution to speed up, secure and scale your API strategy for the digital era. Microcks references any types of APIs or microservices contracts and examples.
- 10. @MICROCKSIO #KAFKASUMMIT Microcks Use-cases API Provider API Provider API Consumer Turn specification into live mocks ! Ensure your implementation is compliant with contract
- 11. #APIDAYS PARIS 2020 @MICROCKSIO Magic Demo Time Photo by Unsplash
- 12. @MICROCKSIO #KAFKASUMMIT All Kafka Every N sec. user-signedup-api-0. 1.1-user-signedup Microcks connected broker (mocking topic generated by Microcks) Microcks connected schema registry Any Schema Registry Listen events during test Any Kafka broker user-registration (real application topic) Azure Event Hubs
- 13. icons by Made x Made from the Noun Project #SOLACE SE @MICROCKSIO EASE COLLABORATION REUSE EXISTING ASSETS CONSISTENT APPROACH REST, ASYNC, SOAP WS Microcks Benefits RUNS EVERYWHERE KUBERNETES IS RUNNING ON THE CLOUD OR ON-PREMISES ACCELERATE FEEDBACKS AND REDUCE TIME TO MARKET SIMULATE ENDPOINTS VALIDATE CONTINUOUSLY
- 14. @MICROCKSIO #KAFKASUMMIT More Resources Apache Kafka Mocking and Testing https://microcks.io/blog/apache-kafka-mocking-testing/ Continuous Testing - OpenAPI & AsyncAPI Together https://microcks.io/blog/continuous-testing-all-your-apis/ How Microcks Can Speed-Up Your AsyncAPI Adoption - Part 1 & 2 https://www.asyncapi.com/blog/microcks-asyncapi-part1 https://www.asyncapi.com/blog/microcks-asyncapi-part2 Join the community on Zulip https://microcksio.zulipchat.com
- 15. @MICROCKSIO #KAFKASUMMIT We Need You! Try Microcks (microcks.io) ! Come chatting on Gitter gitter.im/microcks Follow us on Twitter @microcksio, Distribute & if you like it. It’s a 100% Open Source. And if you love it, contribute !
- 16. @MICROCKSIO #KAFKASUMMIT THANK YOU ! Hugo Guerrero @hguerreroo https://www.youtube.com/user/hguerreroo https://hellokube.dev https://developers.redhat.com/author/hugo-guerrero