The Four Quadrant Model of Monitoring Streaming Data Infrastructure | Praveen Yedidi, Crowdstrike

•

0 likes•429 views

HostedbyConfluent

Kafka plays a vital role in our infrastructure at CrowdStrike. We ingest around 5 trillion events per week into our cloud platform and it is very important that this platform is available, operational, reliable and maintainable. In this talk we will explore how we realised that vision of production readiness at scale by categorising the open source and internal tools we use into 4 quadrants. 1. Observability 2. Availability 3. Operability 4. Data Quality In each quadrant we explore the details of tools used and highlight the critical roles they play in operating our streaming data infrastructure. We start with observability, where we will explore Kafka broker and consumer monitoring tooling. We will take a look at the auto remediations we created to help improve our availability and operability. Finally we will explore how we use message tracing to help us detect message loss and auto recover them to improve data quality.

Four Quadrant model for Monitoring Streaming Data
Infrastructure
PRAVEEN YEDIDI

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

§ Kafka at CrowdStrike
§ Four Quadrant Model
§ Observability
§ Availability & Operability
§ Quality
AGENDA
©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
KAFKA @ CROWDSTRIKE

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
FOUR QUADRANT MODEL

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
OBSERVABILITY

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
ENSURING KAFKA CLUSTER IS OPERATIONAL

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
CONSUMER LAG MONITORING

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
MONITORING LAG WITH BURROW

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
AVAILABILITY

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
SHARDING KAFKA FOR INCREASED AVAILABILITY & RELIABILITY

§ Infinite Scaling
§ Ease of maintenance
§ Fault reliability
©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
BENEFITS

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
ALERT RESPONDER

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
OPERABILITY

§ Scheduled Scaling
§ Scaling based on consumer lag
AUTOSCALING
©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
QUALITY

ADDRESSING UNEVEN PARTITION LAG IN KAFKA
©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
ADDRESSING UNEVEN PARTITION LAG IN KAFKA

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
SLA BASED MONITORING

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
CONCLUSION

©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.
Join us: https://crowdstrike.com/careers
Latest Blogs: https://crowdstrike.com/blog
Thank You

More Related Content

What's hot

Découvrez NGINX AppProtect

Découvrez NGINX AppProtect

Découvrez NGINX AppProtect

This year's first round of Kubernetes and Cloud Native meetups in Eastern Canada began with an update of the CNCF by Ayrat Khayretdinov, CNCF Ambassador and Solutions Architect at CloudOps. He explained the status of various projects and highlights from KubeCon + CloudNativeCon. To learn the basics of cloud native application modernization, sign up for one of our hands-on, three-day workshops on Docker and Kubernetes at https://www.cloudops.com/workshops/#DockerK8s

Kubernetes and Cloud Native Meetup - March, 2019

Kubernetes and Cloud Native Meetup - March, 2019

Kubernetes and Cloud Native Meetup - March, 2019

********** Watch this presentation on-demand! https://info.lightbend.com/iot-megaservices-high-throughput-microservices-with-akka-register.html ********** In this interactive presentation by Hugh McKee, Developer Advocate at Lightbend, we’ll share our experiences helping our clients create a system architecture that can support high throughput microservices (aka "Megaservices"). We’ll do that using IoT demo applications designed to push cloud service providers like Amazon and Google to their limits. Using sample code that you can later run on your own machine, we’ll look at: * Modeling real-life digital twins for hundreds of thousands of IoT devices in the field, looking into how these megaservices are implemented in Akka. * Visualizing Akka Actors–which represent IoT digital twins–in a “crop circle” formation that represents a complete distributed Reactive application, and watching at messages are processed across Akka Cluster nodes using cluster sharding. * Some code behind the whole set up, which is built using OSS like Akka, Java, JavaScript, and Kubernetes. Follow us on social: TW: https://twitter.com/lightbend LI: https://www.linkedin.com/company/lightbend-inc-/ FB: https://www.facebook.com/lightbendOfficial/ For more about Lightbend: Blog: https://www.lightbend.com/blog  Newsletter: https://www.lightbend.com/newsletter 

IoT 'Megaservices' - High Throughput Microservices with Akka

IoT 'Megaservices' - High Throughput Microservices with Akka

IoT 'Megaservices' - High Throughput Microservices with Akka

On-Demand Recording https://www.nginx.com/resources/webinars/control-kubernetes-ingress-egress-together-nginx/ About the Webinar Join our resident Kubernetes and modern apps experts in a discussion of the challenges of Kubernetes traffic management in today’s technology landscape. While Kubernetes Ingress gets most of the attention, how you handle egress traffic is just as important. Egress isn’t just about traffic leaving a cluster, either, but also concerns traffic among managed and unmanaged services within the cluster. We demo a solution using NGINX Service Mesh and NGINX Ingress Controller to control egress from the cluster and between NGINX Service Mesh and unmanaged services. Whether you’re new to modern application architectures, or looking to improve your current microservices deployment, this webinar is for you. Speakers: Amir Rawdat Technical Marketing Engineer F5 Faisal Memon Software Engineer F5

Control Kubernetes Ingress and Egress Together with NGINX

Control Kubernetes Ingress and Egress Together with NGINX

Control Kubernetes Ingress and Egress Together with NGINX

About the Webinar App developers and DevOps teams at more than 400 million websites rely on NGINX for a wide range of app delivery functions – load balancer, API gateway, reverse proxy, and web server. But the more NGINX instances you have, the bigger the challenge in keeping track of them all. That’s where NGINX Instance Manager comes in. Join this webinar to learn about a new solution called NGINX Instance Manager that helps you to effortlessly discover, configure, and monitor NGINX Open Source and NGINX Plus instances at scale. The webinar includes a demo of NGINX Instance Manager in action. Tom Gamull Product Manager, F5 Karthik Krishnaswamy Director, Product Marketing, NGINX

Nim tames sprawl

Nim tames sprawl

Nim tames sprawl

Nous vous présenterons la solution NGINX Controller facilitant vos workflows CI/CD. Vous découvrirez la gestion de plusieurs instances NGINX, de répartiteurs de charges, de passerelles API, de métriques dans des environnements applicatifs hétérogènes depuis une console unique. Join this webinar to learn: - Accélérer la mise sur le marché sans compromettre la sécurité et les performances - Comment accélérer le déploiement de vos applications - Comment la solution de management NGINX Controller facilite - - l’intégration d’un workflow CI/CD de dernière génération - L’agilité de NGINX Plus On-Demand Link: https://www.nginx.com/resources/webinars/accelerez-vos-deploiements-applicatifs-avec-nginx-controller/

Accélérez vos déploiements applicatifs avec NGINX Controller

Accélérez vos déploiements applicatifs avec NGINX Controller

Accélérez vos déploiements applicatifs avec NGINX Controller

Implementing MySQL Database-as-a-Service using open source tools

Implementing MySQL Database-as-a-Service using open source tools

Implementing MySQL Database-as-a-Service using open source tools

All Things Open

See how open vCPE can be achieved in the real world and in action, while integrating other VNFs into the service chain, while easily instantiating and managing on any cloud, leveraging open orchestration design concepts. More and more vendors are looking to not only easily onboard their VNFs to the cloud, but also build a stack that is versatile and not locked into one cloud provider or vendor. Join this webinar and learn how Datavision and Cloudify are helping deliver this end-to-end solution across the globe

Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration

Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration

Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration

Cloudify Community

Managing traffic routing with istio and envoy workshop

Managing traffic routing with istio and envoy workshop

Managing traffic routing with istio and envoy workshop

Cloud Economics - Crayon Optimization Services

Cloud Economics - Crayon Optimization Services

Cloud Economics - Crayon Optimization Services

Convincing developers to write tests for new code is hard; convincing developers to write tests for new event data is even harder. At Reddit, engineers have often deployed new app versions, only to find out later that the event wasn’t firing at all, or it was missing critical fields. So this begs the question, “How can engineers at Reddit be confident that the events they instrument are accurate and complete?” In this session, we will learn about an internal tool developed at Reddit to QA events in real-time. This KSQL-powered web app streams events from our pipeline, allowing developers to filter events they care about using criteria like User ID, Device ID or the type of user interaction. With a backbone of KSQL and Kafka Streams, engineers can get real-time feedback on how accurate (or how erroneous) their event data is.

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...

HostedbyConfluent

Speakers: Alan Murphy, Regional Solution Architect for APAC NGINX, Inc. Karthik Krishnaswamy, Senior Product Marketing Manager at NGINX, Inc. About the webinar Deploying and managing applications at scale always presents challenges. Infrastructure has to be configured and provisioned quickly across multiple public and private clouds. There’s ever-increasing pressure to maintain reliability, performance, and availability of business applications across this disparate infrastructure. Easy workflows for infrastructure and DevOps, as well as self-service management capabilities for developers, are needed to accelerate application deployment. The result? Enterprises are able to innovate and bring apps to market faster. That’s where NGINX Controller comes in. Controller helps users manage multiple NGINX instances at scale with centralized configuration, monitoring, alerting, and troubleshooting.

NGINX Controller: Configuration, Management, and Troubleshooting at Scale

NGINX Controller: Configuration, Management, and Troubleshooting at Scale

NGINX Controller: Configuration, Management, and Troubleshooting at Scale

Services are at the core of modern software architecture. Deploying a series of modular, small (micro-)services rather than big monoliths gives developers the flexibility to work in different languages, technologies and release cadence across the system; resulting in higher productivity and velocity, especially for larger teams. With the adoption of microservices, however, new problems emerge due to the sheer number of services that exist in a larger system. Problems that had to be solved once for a monolith, like security, load balancing, monitoring, and rate limiting need to be handled for each service. Istio, announced at GlueCon 2017, addresses these problems in a fundamental way through a service mesh framework. With Istio, developers can implement the core logic for the microservices, and let the framework take care of the rest – traffic management, discovery, service identity and security, and policy enforcement. Better yet, this can be also done for existing microservices without rewriting or recompiling any of their parts. Istio uses Envoy as its runtime proxy component and provides an extensible intermediation layer which allows global cross-cutting policy enforcement and telemetry collection.

An Open-Source Platform to Connect, Manage, and Secure Microservices

An Open-Source Platform to Connect, Manage, and Secure Microservices

An Open-Source Platform to Connect, Manage, and Secure Microservices

DoiT International

What's New with NGINX Application Security Solutions

What's New with NGINX Application Security Solutions

What's New with NGINX Application Security Solutions

Kubernetes Networking

Kubernetes Networking

Kubernetes Networking

(Vahid Fereydouny, Confluent) Kafka Summit SF 2018 Security and compliance are key concerns for many organizations today and it is very important that we can meet these requirements in our platform. This is also extremely critical for customers who are adopting Confluent cloud offerings, since moving the streaming platform to cloud exposes new security and governance issues. In this session, we will discuss how Confluent is providing control and visibility to address these concerns and enable secure streaming platforms. We will cover the main pillars of IT security in access control (authentication, authorization), data confidentiality (encryption) and auditing.

End-End Security with Confluent Platform

End-End Security with Confluent Platform

End-End Security with Confluent Platform

NGINX: Back to Basics – APCJ

NGINX: Back to Basics – APCJ

NGINX: Back to Basics – APCJ

Production-Grade Kubernetes With NGINX Ingress Controller

Production-Grade Kubernetes With NGINX Ingress Controller

Production-Grade Kubernetes With NGINX Ingress Controller

About the webinar The use of an API gateway and the move to microservices are two of the most important trends in application development. But are they similar, or different; complementary, or contradictory? In this webinar, we discuss the advantages of an API gateway, the advantages of microservices development, and how and when they can work together. The NGINX Microservices Reference Architecture (MRA) uses three different network architectures, with service mesh as a fourth. We describe how an API gateway relates to each of these network architectures and how to reduce rework if your application needs to evolve from one architecture to another. Speakers: Charles Pretzer, Technical Architect, NGINX, Inc. Floyd Smith, Director of Content Marketing, NGINX, Inc.

Using an API Gateway for Microservices

Using an API Gateway for Microservices

Using an API Gateway for Microservices

Speaker: Karthik Krishnaswamy, Sr Product Marketing Manager NGINX, Inc. About the Webinar Technology is revolutionizing our lives – we hail cabs, order food, and stream video right from apps on our mobile devices. At the heart of this disruption are enterprises that embrace digital transformation – according to a survey from IDC sponsored by NGINX, 75% of businesses will have completed their digital transformation by 2030. To support digital transformation, enterprises are modernizing their applications and underlying infrastructure by adopting a variety of processes and new technologies such as DevOps, cloud, open source, and microservices. Software load balancing is central to accelerating digital transformation efforts. That’s where NGINX comes in – it’s a lightweight, flexible and portable load balancer that helps you gain agility and simplifies your architecture In this webinar we cover how to gracefully migrate your BIG-IP deployment to NGINX, the ADC built for modern applications.

Migrating from BIG-IP Deployment to NGINX ADC

Migrating from BIG-IP Deployment to NGINX ADC

Migrating from BIG-IP Deployment to NGINX ADC

What's hot (20)

Découvrez NGINX AppProtect

Découvrez NGINX AppProtect

Découvrez NGINX AppProtect

Kubernetes and Cloud Native Meetup - March, 2019

Kubernetes and Cloud Native Meetup - March, 2019

Kubernetes and Cloud Native Meetup - March, 2019

IoT 'Megaservices' - High Throughput Microservices with Akka

IoT 'Megaservices' - High Throughput Microservices with Akka

IoT 'Megaservices' - High Throughput Microservices with Akka

Control Kubernetes Ingress and Egress Together with NGINX

Control Kubernetes Ingress and Egress Together with NGINX

Control Kubernetes Ingress and Egress Together with NGINX

Nim tames sprawl

Nim tames sprawl

Nim tames sprawl

Accélérez vos déploiements applicatifs avec NGINX Controller

Accélérez vos déploiements applicatifs avec NGINX Controller

Accélérez vos déploiements applicatifs avec NGINX Controller

Implementing MySQL Database-as-a-Service using open source tools

Implementing MySQL Database-as-a-Service using open source tools

Implementing MySQL Database-as-a-Service using open source tools

Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration

Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration

Cloudify: Open vCPE Design Concepts and Multi-Cloud Orchestration

Managing traffic routing with istio and envoy workshop

Managing traffic routing with istio and envoy workshop

Managing traffic routing with istio and envoy workshop

Cloud Economics - Crayon Optimization Services

Cloud Economics - Crayon Optimization Services

Cloud Economics - Crayon Optimization Services

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...

NGINX Controller: Configuration, Management, and Troubleshooting at Scale

NGINX Controller: Configuration, Management, and Troubleshooting at Scale

NGINX Controller: Configuration, Management, and Troubleshooting at Scale

An Open-Source Platform to Connect, Manage, and Secure Microservices

An Open-Source Platform to Connect, Manage, and Secure Microservices

An Open-Source Platform to Connect, Manage, and Secure Microservices

What's New with NGINX Application Security Solutions

What's New with NGINX Application Security Solutions

What's New with NGINX Application Security Solutions

Kubernetes Networking

Kubernetes Networking

Kubernetes Networking

End-End Security with Confluent Platform

End-End Security with Confluent Platform

End-End Security with Confluent Platform

NGINX: Back to Basics – APCJ

NGINX: Back to Basics – APCJ

NGINX: Back to Basics – APCJ

Production-Grade Kubernetes With NGINX Ingress Controller

Production-Grade Kubernetes With NGINX Ingress Controller

Production-Grade Kubernetes With NGINX Ingress Controller

Using an API Gateway for Microservices

Using an API Gateway for Microservices

Using an API Gateway for Microservices

Migrating from BIG-IP Deployment to NGINX ADC

Migrating from BIG-IP Deployment to NGINX ADC

Migrating from BIG-IP Deployment to NGINX ADC

Similar to The Four Quadrant Model of Monitoring Streaming Data Infrastructure | Praveen Yedidi, Crowdstrike

In this talk, we look at textbook examples of using kafka at scale. Specifically focused on Evernorth Health Service's journey of implementing microservices data pipelines, we provide an overview of the patterns we used while implementing CDC data pipelines for these micro services using confluent kafka, the challenges we faced, lessons learned and the unique solutions that we developed over the years to overcome those challenges. We also look at how we are moving these micro services eco systems to public cloud (AWS) and the strategies that we have/are implementing to ensure a smooth consumer cutover. We peek into how kafka consumers can cutover to a replicated topic using offsets based on create time timestamps and how/why this is critical for a downtime-less and data-loss-less consumer cutover for streaming consumers. To conclude, we take a look at how we are re-imagining these pipelines on AWS and how SaaS offerings like Confluent Cloud and Confluent connectors could play a major role.

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...

HostedbyConfluent

Hands-On Lab: CA Spectrum® 10.0 Deep Dive – 64-Bit, Network Virtualization an...

Hands-On Lab: CA Spectrum® 10.0 Deep Dive – 64-Bit, Network Virtualization an...

Hands-On Lab: CA Spectrum® 10.0 Deep Dive – 64-Bit, Network Virtualization an...

CA Technologies

Kong Mesh入門編

Companies across all industries are innovating with software to stay competitive, connect with customers, grow new revenue sources, and transform their business. As an IT operations or applications leader, you need to leverage your VMware investments to innovate faster to deliver applications in weeks, not months. Pivotal CF, the leading enterprise Platform-as-a-Service, powered by Cloud Foundry, enables IT operations teams to do just that: accelerate software delivery on their vSphere-based private clouds, and on VMware’s public cloud, vCloud Air. With Pivotal CF, you can simultaneously improve developer productivity while gaining huge operational efficiencies.

Supercharge Your Application Delivery

Supercharge Your Application Delivery

Supercharge Your Application Delivery

Freeing the World from Slow: How Service Virtualization and the Concept of S....

Freeing the World from Slow: How Service Virtualization and the Concept of S....

Freeing the World from Slow: How Service Virtualization and the Concept of S....

CA Technologies

Using OpenContrail with Kubernetes

Using OpenContrail with Kubernetes

Using OpenContrail with Kubernetes

What Is Spring?

What Is Spring?

What Is Spring?

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

Welcome to the application economy, where every company is a software company. You are challenged to build applications faster, cheaper and better, but can't seem to achieve all three goals at the same time. In this presentation, you will learn where the bottlenecks occur in application development and how you spot defects earlier in the process, automate your testing and build better apps. CA Continuous Application Insight provides deep transaction insight into your Web and mobile applications, allowing developers to identify issues much earlier, make changes and produce cleaner code. The result is a better app and a dramatically improved customer experience. For more information, please visit http://cainc.to/Nv2VOe

Technology Primer: Building Applications the New-Fashioned Way

Technology Primer: Building Applications the New-Fashioned Way

Technology Primer: Building Applications the New-Fashioned Way

CA Technologies

Microservices & Containers - Docker Bangalore Meetup #12

Microservices & Containers - Docker Bangalore Meetup #12

Microservices & Containers - Docker Bangalore Meetup #12

Companies that understand how to overcome Data Gravity with a LiveData strategy stand to benefit the most from innovations in data and analytics. Others will continue to invest in legacy systems and data silos that fail to take advantage of distributed computing and the rapid pace of data product innovation on the cloud. This talk shares best practices our clients have taken to evolve their data architecture to become a LiveData company.

Strata London WANdisco

Strata London WANdisco

Strata London WANdisco

Challenges of Being an ISV and Customer in a K8s World

Challenges of Being an ISV and Customer in a K8s World

Challenges of Being an ISV and Customer in a K8s World

Before the Web, clients connected to back-end servers without compromises. As the Web came along with HTTP, we had to give up some of the power of our connectivity. With the evolving open Web communications standards, now we have a unique opportunity to move on from a request-response based REST world to a true, event-driven architecture. Learn what it takes to connect physical devices, such as Arduino and the Rasberry Pi to collect sensor data, or control physical devices over the Web. Commercial aviation's biggest challenge when things go wrong is that flight data is stuck on the plane inside the black box. Until the black box is recovered, we barely know anything about what went wrong. This presentation offers detailed insights into how IoT and modern Web communications concepts have the power to change all this. If you're a maker, you will learn about "flight sensors" attached and controlled by Arduino, data transmitted over long range WiFi as well as satellite networks. If interested in real-time Web communications, you'll learn about a highly secure WebSocket implementation with extreme scale, publishing flight data to tablets and laptops used as monitoring dashboard. If protocol layering, enterprise messaging, or JMS is your thing, this talk is for you. Code samples and plenty of interactive live demos (with Things) add color to the talk.

Connecting Physical Devices to the Web - Event Driven Architecture using WebS...

Connecting Physical Devices to the Web - Event Driven Architecture using WebS...

Connecting Physical Devices to the Web - Event Driven Architecture using WebS...

Peter Moskovits

Companies across all industries are innovating with software to stay competitive, connect with customers, grow new revenue sources, and transform their business. As an IT operations or applications leader, you need to leverage your VMware investments to innovate faster to deliver applications in weeks, not months. Pivotal CF, the leading enterprise Platform-as-a-Service, powered by Cloud Foundry, enables IT operations teams to do just that: accelerate software delivery on their vSphere-based private clouds, and on VMware’s public cloud, vCloud Air. With Pivotal CF, you can simultaneously improve developer productivity while gaining huge operational efficiencies.

Supercharge Your Application Delivery: The Journey to Enterprise PaaS

Supercharge Your Application Delivery: The Journey to Enterprise PaaS

Supercharge Your Application Delivery: The Journey to Enterprise PaaS

Pre-Con Ed: CA API Gateway: Managing and Migrating Policies with the Gateway ...

Pre-Con Ed: CA API Gateway: Managing and Migrating Policies with the Gateway ...

Pre-Con Ed: CA API Gateway: Managing and Migrating Policies with the Gateway ...

CA Technologies

Join the discussion with NetApp and CA Technologies Storage Experts and learn how you can store more, spend less and increase business agility – yes, it’s possible! Discover how flexible on-premise and cloud innovations are helping organizations lower operating costs and reduce administrative overhead associated with managing and scaling z Systems storage. Learn how you can accelerate business responsiveness by using the cloud to efficiently store, access, retrieve and recover z/OS data. Join us for an informative discussion and hear about new features and strategic plans that can help your business reduce the total cost of ownership (TCO) for storing and managing your mainframe data and increase your overall efficiency. For more information, please visit http://cainc.to/Nv2VOe

Driving Down Costs of z Systems™ Storage

Driving Down Costs of z Systems™ Storage

Driving Down Costs of z Systems™ Storage

CA Technologies

Implementing Adaptable Microservices; A Methodology for Loosely-Coupled Compo...

Implementing Adaptable Microservices; A Methodology for Loosely-Coupled Compo...

Implementing Adaptable Microservices; A Methodology for Loosely-Coupled Compo...

CA Technologies

Discussion on handling WebRTC media: -What are the main reasons to terminate WebRTC media -Media server use cases in WebRTC -Client-side media processing vs. server-side trade-offs -Potential media services for today & tomorrow -Introduction to the Media Resource Broker (MRB) for scaling and orchestrating media servers/MRF -How media handling architectures are evolving & scaling for cloud/NFV networks

WebRTC Infrastructure the Hard Parts: Media

WebRTC Infrastructure the Hard Parts: Media

WebRTC Infrastructure the Hard Parts: Media

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

Couchbase Cloud No Equal (Rick Jacobs, Couchbase) Kafka Summit 2020

Couchbase Cloud No Equal (Rick Jacobs, Couchbase) Kafka Summit 2020

Couchbase Cloud No Equal (Rick Jacobs, Couchbase) Kafka Summit 2020

HostedbyConfluent

Similar to The Four Quadrant Model of Monitoring Streaming Data Infrastructure | Praveen Yedidi, Crowdstrike (20)

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...

Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...

Hands-On Lab: CA Spectrum® 10.0 Deep Dive – 64-Bit, Network Virtualization an...

Hands-On Lab: CA Spectrum® 10.0 Deep Dive – 64-Bit, Network Virtualization an...

Hands-On Lab: CA Spectrum® 10.0 Deep Dive – 64-Bit, Network Virtualization an...

Kong Mesh入門編

Supercharge Your Application Delivery

Supercharge Your Application Delivery

Supercharge Your Application Delivery

Freeing the World from Slow: How Service Virtualization and the Concept of S....

Freeing the World from Slow: How Service Virtualization and the Concept of S....

Freeing the World from Slow: How Service Virtualization and the Concept of S....

Using OpenContrail with Kubernetes

Using OpenContrail with Kubernetes

Using OpenContrail with Kubernetes

What Is Spring?

What Is Spring?

What Is Spring?

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

Paul Dix [InfluxData] | InfluxDays Keynote: Future of InfluxDB | InfluxDays N...

Technology Primer: Building Applications the New-Fashioned Way

Technology Primer: Building Applications the New-Fashioned Way

Technology Primer: Building Applications the New-Fashioned Way

Microservices & Containers - Docker Bangalore Meetup #12

Microservices & Containers - Docker Bangalore Meetup #12

Microservices & Containers - Docker Bangalore Meetup #12

Strata London WANdisco

Strata London WANdisco

Strata London WANdisco

Challenges of Being an ISV and Customer in a K8s World

Challenges of Being an ISV and Customer in a K8s World

Challenges of Being an ISV and Customer in a K8s World

Connecting Physical Devices to the Web - Event Driven Architecture using WebS...

Connecting Physical Devices to the Web - Event Driven Architecture using WebS...

Connecting Physical Devices to the Web - Event Driven Architecture using WebS...

Supercharge Your Application Delivery: The Journey to Enterprise PaaS

Supercharge Your Application Delivery: The Journey to Enterprise PaaS

Supercharge Your Application Delivery: The Journey to Enterprise PaaS

Pre-Con Ed: CA API Gateway: Managing and Migrating Policies with the Gateway ...

Pre-Con Ed: CA API Gateway: Managing and Migrating Policies with the Gateway ...

Pre-Con Ed: CA API Gateway: Managing and Migrating Policies with the Gateway ...

Driving Down Costs of z Systems™ Storage

Driving Down Costs of z Systems™ Storage

Driving Down Costs of z Systems™ Storage

Implementing Adaptable Microservices; A Methodology for Loosely-Coupled Compo...

Implementing Adaptable Microservices; A Methodology for Loosely-Coupled Compo...

Implementing Adaptable Microservices; A Methodology for Loosely-Coupled Compo...

WebRTC Infrastructure the Hard Parts: Media

WebRTC Infrastructure the Hard Parts: Media

WebRTC Infrastructure the Hard Parts: Media

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes

Couchbase Cloud No Equal (Rick Jacobs, Couchbase) Kafka Summit 2020

Couchbase Cloud No Equal (Rick Jacobs, Couchbase) Kafka Summit 2020

Couchbase Cloud No Equal (Rick Jacobs, Couchbase) Kafka Summit 2020

More from HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

Simplifying Mobile A11y Presentation.pptx

Simplifying Mobile A11y Presentation.pptx

Simplifying Mobile A11y Presentation.pptx

The presentation was made in “Web3 Fusion: Embracing AI and Beyond” is more than a conference; it's a journey into the heart of digital transformation. The conference a provided a platform where the future of technology meets practical application. This three-day hybrid event, set in the heart of innovation, served as a gateway to the latest trends and transformative discussions in AI, Blockchain, IoT, AR/VR, and their collective impact on the information space.

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

In this session, we will discuss the journey of API governance from its initial, ungoverned state to the development of sophisticated models that tackle contemporary challenges. We'll explore how APIs have become essential in the intersection of business and technology, adapting to advancements and evolving needs. We'll focus on how organizations have moved from launching to monetizing APIs, using models like pay-per-use and subscriptions, and finding the right balance between technical implementation and business strategy. We'll also highlight the impact of governance on monetization strategies, especially how data security, compliance, and service quality influence pricing. Real-world examples will demonstrate the effective integration of governance with monetization, including AI's role in dynamic pricing. Looking ahead, we'll share insights into future trends in API governance and monetization, emphasizing the importance of adaptability, continuous learning, and innovation.

API Governance and Monetization - The evolution of API governance

API Governance and Monetization - The evolution of API governance

API Governance and Monetization - The evolution of API governance

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

johnbeverley2021

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

caitlingebhard1

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

Platformless Horizons for Digital Adaptability

Platformless Horizons for Digital Adaptability

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Bhuvaneswari Subramani

Explore the latest trends and insights on JavaScript usage with Pixlogix's informative blog. Discover key statistics and facts about JavaScript's role in web development, its popularity among developers, and its impact on modern websites. Stay updated with the evolving landscape of JavaScript frameworks and libraries, and learn how they're shaping the future of web development. Gain valuable insights to enhance your JavaScript skills and stay ahead in the digital realm.

JavaScript Usage Statistics 2024 - The Ultimate Guide

JavaScript Usage Statistics 2024 - The Ultimate Guide

JavaScript Usage Statistics 2024 - The Ultimate Guide

Pixlogix Infotech

Design and Development of a Provenance Capture Platform for Data Science

Design and Development of a Provenance Capture Platform for Data Science

Design and Development of a Provenance Capture Platform for Data Science

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Rustici Software

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

Quantum Leap in Next-Generation Computing

Quantum Leap in Next-Generation Computing

Quantum Leap in Next-Generation Computing

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

rightmanforbloodline

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Recently uploaded (20)

Simplifying Mobile A11y Presentation.pptx

Simplifying Mobile A11y Presentation.pptx

Simplifying Mobile A11y Presentation.pptx

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

AI in Action: Real World Use Cases by Anitaraj

API Governance and Monetization - The evolution of API governance

API Governance and Monetization - The evolution of API governance

API Governance and Monetization - The evolution of API governance

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

Six Myths about Ontologies: The Basics of Formal Ontology

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

Stronger Together: Developing an Organizational Strategy for Accessible Desig...

Platformless Horizons for Digital Adaptability

Platformless Horizons for Digital Adaptability

Platformless Horizons for Digital Adaptability

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Elevate Developer Efficiency & build GenAI Application with Amazon Q

JavaScript Usage Statistics 2024 - The Ultimate Guide

JavaScript Usage Statistics 2024 - The Ultimate Guide

JavaScript Usage Statistics 2024 - The Ultimate Guide

Design and Development of a Provenance Capture Platform for Data Science

Design and Development of a Provenance Capture Platform for Data Science

Design and Development of a Provenance Capture Platform for Data Science

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

MINDCTI Revenue Release Quarter One 2024

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

Corporate and higher education May webinar.pptx

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2's API Vision: Unifying Control, Empowering Developers

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)

Quantum Leap in Next-Generation Computing

Quantum Leap in Next-Generation Computing

Quantum Leap in Next-Generation Computing

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

Exploring Multimodal Embeddings with Milvus

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

DBX First Quarter 2024 Investor Presentation

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

The Four Quadrant Model of Monitoring Streaming Data Infrastructure | Praveen Yedidi, Crowdstrike

1. Four Quadrant model for Monitoring Streaming Data Infrastructure PRAVEEN YEDIDI

2. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

3. § Kafka at CrowdStrike § Four Quadrant Model § Observability § Availability & Operability § Quality AGENDA ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

4. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. KAFKA @ CROWDSTRIKE

5. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. FOUR QUADRANT MODEL

6. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. OBSERVABILITY

7. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. ENSURING KAFKA CLUSTER IS OPERATIONAL

8. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. CONSUMER LAG MONITORING

9. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. MONITORING LAG WITH BURROW

10. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. AVAILABILITY

11. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. SHARDING KAFKA FOR INCREASED AVAILABILITY & RELIABILITY

12. § Infinite Scaling § Ease of maintenance § Fault reliability ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. BENEFITS

13. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. ALERT RESPONDER

14. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. OPERABILITY

15. § Scheduled Scaling § Scaling based on consumer lag AUTOSCALING ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

16. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. QUALITY

17. ADDRESSING UNEVEN PARTITION LAG IN KAFKA ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED.

18. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. ADDRESSING UNEVEN PARTITION LAG IN KAFKA

19. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. SLA BASED MONITORING

20. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. CONCLUSION

21. ©2021 CROWDSTRIKE, INC. ALL RIGHTS RESERVED. Join us: https://crowdstrike.com/careers Latest Blogs: https://crowdstrike.com/blog Thank You