Lessons from the field: Catalog of Kafka Deployments | Joseph Niemiec, Cloudera

•

0 likes•386 views

Streaming architectures have been on the rise steadily and as a result, we have seen the adoption of Kafka go up too. With the diverse spread of use cases across multiple industries, we have seen a variety of Kafka deployments across our hundreds of Kafka customers. Along the way, we have learnt some best practices as well as what not to do in mission-critical architectures. Join Joe Niemiec, Sr. Product Manager at Cloudera, as he shares these insights in this session that covers topics such as - The many ways that Kafka has been deployed in the field Standalone clusters, multiple clusters in a single data center and multiple clusters geographically distributed performing replication Clusters of all sizes small and large, few messages to hundreds of thousands per second Discussion about architecture failure domains Configurations tuned and used in specific deployments

Technology

Lessons from the field:
Catalog of Kafka Deployments
Joseph Niemiec | Sr. Product Manager

3
© 2021 Cloudera, Inc. All rights reserved.
Failure/Fault Domains
Limiting the state and scope of failure
What is a Failure/Fault Domain?
A set of components that share a
single point of failure and fail in a
correlated manner

4
© 2021 Cloudera, Inc. All rights reserved.
Serial and Parallel Systems Reliability
Serial Systems
• Failure of any component results
in failure of entire system
• Adding more weakens the
system ie - 90% * 90% = 81%
– Rs= R1* R2… Rn
Parallel Systems
• One component online results in
system online
• Adding more strengthens the system
ie - 1-(1-0.90)*(1-0.90) = 99%
– Rs= 1 - (1 -R1) *(1-R2) … (1-Rn)

5
© 2021 Cloudera, Inc. All rights reserved.
Basic Topic Design Theory
Partitions
● Few as needed
● Increased open files pressure
Topic Replicas
● Parallel availability
min-insync.replicas
● Durability vs Availability
● ack=all
○ for all isr with at least the min isr available
● replica.lag.time.max.ms
○ default 30 seconds!!!
Use log.retention.bytes
● Can use together
● Protect from topics growing larger
than brokers
Log Segments need to be
eligible for retention policies
● log.segment.bytes and log.roll.ms

7
© 2021 Cloudera, Inc. All rights reserved.
Single Node Dev Cluster
• Everything is Serial
• Failure/Fault Domain
– The Cluster
– The Broker & Zookeeper
– The Laptop
– Everything!
• Failure/Fault Tolerance
– Deploy over Multiple Servers
Kafka Cluster
1 Broker
1 Zookeeper

8
© 2021 Cloudera, Inc. All rights reserved.
Single Cluster - Single Rack w/wo colocated Zookeeper
• Some things are Serial
• Some things are Parallel
• Failure/Fault Domain
– Rack
– Brokers
• Other Notes
– Shared Log Dirs for Brokers
and Zookeeper

9
© 2021 Cloudera, Inc. All rights reserved.
Single Cluster - VMs
• Some things are Serial
• Some things are Parallel
• Failure/Fault Domain
– VM Hosts
– Replica Placement
• Replica per VM Host
• Other Notes
– Debugging Complexity

10
© 2021 Cloudera, Inc. All rights reserved.
Classic Failover - Two Clusters AcFve / Passive
• Mostly Parallel*
• Failure/Fault Domain
– Mirror Makers
– Cluster
– WAN*
– Data Center
• Other Notes
– Offsets During Failover
– WAN Bandwidth
– Mirror Maker Producer Side

11
© 2021 Cloudera, Inc. All rights reserved.
STREAMS
REPLICATION
MANAGER (SRM)
with Mirror Maker2
• Supports active-active, multi-
cluster, cross DC replication &
other scenarios
• Leverage Kafka Connect for
scalability and HA
• Replicate data and configurations
• Offset translation for easy
failover

12
© 2021 Cloudera, Inc. All rights reserved.
SRM
MONITORING
SERVICES
• SMM Cluster Replications View
provides monitoring
• SRM Service calculates replication
metrics (latency, throughput)
• REST endpoints with Swagger UI
• SMM Replication Flow Alerting

13
© 2021 Cloudera, Inc. All rights reserved.
Triple OnSite Cluster - In/Out/SysOps
• Serial Pipeline
• Parallel Systems
• Failure/Fault Domain
– Cluster
• Isolated Failures in Pipeline
• IN / Out Pipeline
– Producers / Apps / Consumers
– Datacenter
• Other Notes
– Test Configs/Apps on DevOps
– Ingress / Egress Isolation

14
© 2021 Cloudera, Inc. All rights reserved.
MulF-Geo DistribuFon and AggregaFon
• Serial Pipeline
• Parallel Systems
• Failure/Fault Domain
– Data Center
– Mirror Maker
– Cluster
• Other Notes
– Fat Network Pipe

15
© 2021 Cloudera, Inc. All rights reserved.
Dual Aggregation Active Active
• Mostly Parallel*
• Failure/Fault Domain
– Data Center
– Cluster
– Mirror Maker
– Consumers / Producers*
• Other Notes
– Active/Active
– Ingress / Egress Isolation
– Consumers on any side

16
© 2021 Cloudera, Inc. All rights reserved.
Multi-AZ/Cloud Spanning
• Parallel*
• Failure/Fault Domain
– Availability Zone
– Replica Placement*
• Other Notes
– Rack Awareness
– min-insync.replicas

This session will dive into our most successful (and unsuccessful!) multi-cluster event replication patterns. An x-ray of the cross cluster distribution model that powers our globally distributed APIs will touch on the benefits that this model has provided in terms of client API experience, delivery agility and developer experience. We will focus on recipes for effective use of Mirror Maker event replication to power platform distribution including the challenges of managing a 'fan in' event replication workflow - pulling events created in satellite clusters back to a mothership cluster for processing. We will introduce the elegant technique of replication event multiplexing - which can be used to simplify the burden of managing a 'fan-in' replication topology by eliminating regional awareness from the application domain and improving replication health monitoring & observability.

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

HostedbyConfluent

You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

HostedbyConfluent

Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

HostedbyConfluent

The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.

Kubernetes connectivity to Cloud Native Kafka | Evan Shortiss and Hugo Guerre...

HostedbyConfluent

If you want to build an ecosystem of streaming data to your Kafka platform, you will need a much easier way for your developer to quickly move what’s on the source to your cluster. Better yet, making the connector serverless so it would NOT waste any resources for being idle, and having a trusted partner manage your Kafka infrastructure for you. In this session, we will show you how easy we have made streaming data with great user experience. Flexible resource management with our new secret weapon in the Apache Camel project -- Kamelet. We’ll also demonstrate how Red Hat OpenShift Streams for Apache Kafka simplifies the provisioning of Kafka deployments in a public cloud, managing the cluster,topics, and configuring secure access to the Kafka cluster for your developers.

Availability of Kafka - Beyond the Brokers | Andrew Borley and Emma Humber, IBM

HostedbyConfluent

While Kafka has guarantees around the number of server failures a cluster can tolerate, to avoid service interruptions, or even data loss, it is prudent to have infrastructure in place for when an environment becomes unavailable during a planned or unplanned outage. This talk describes the architectures available to you when planning for an outage. We will examine configurations including active/passive and active/active as well as availability zones and debate the benefits and limitations of each. We will also cover how to set up each configuration using the tools in Kafka. Whether downtime while you fail over clients to a backup is acceptable or you require your Kafka clusters to be highly available, this talk will give you an understanding of the options available to mitigate the impact of the loss of an environment.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | J...

HostedbyConfluent

As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Apache Kafka. We believe that CIP positions us for the best defense against cyber threats well into the future. Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It provides us the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately leads to better prevention. In our session, we’ll discuss the details described in the IT@Intel white paper that was published in Nov 2020 with same title.

Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff ...

HostedbyConfluent

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

HostedbyConfluent

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. Kafka was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...

HostedbyConfluent

Many organizations use Apache Kafka to facilitate the flow of data between multiple applications or data sources. Thanks to Kafka’s distributed architecture, it is easy to set up a scalable and reliable broker, but doing the same with producers or consumers is quite often a fine art. This session provides a quick overview of Couchbase, describes the Couchbase Kafka Connector, and showcases a demo of how it can be used as both a source and a sink for building real-time data processing pipelines for mission-critical applications.

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

HostedbyConfluent

Do your event streams use connected-data domains such as fraud detection, live logistics routing, or predicting network outages? How can you maintain the analysis and leverage those connections real-time? Graph databases differ from traditional, tabular ones in that they treat connections between data as first class citizens. This means they are optimized for detecting and understanding these relationships – providing insight at speed and at scale. By combining event streams from Kafka along with the power of the Neo4j graph database for interrogating and investigating connections, you make real-time, event-driven intelligent insight a reality. Neo4j Streams integrates Neo4j with Apache Kafka event streams, to serve as a source of data, for instance Change Data Capture or a sink to ingest any kind of Kafka event into your graph. In this session we’ll show you how to get up and running with Neo4j Streams to show you how to sink and source between graphs and streams.

Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset

HostedbyConfluent

Streaming data systems have been growing rapidly in importance to the modern data stack. Kafka’s kSQL provides an interface for analytic tools that speak SQL. Apache Superset, the most popular modern open-source visualization and analytics solution, plugs into nearly any data source that speaks SQL, including Kafka. Here, we review and compare methods for connecting Kafka to Superset to enable streaming analytics use cases including anomaly detection, operational monitoring, and online data integration.

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

HostedbyConfluent

Cloud is changing the world; Kubernetes is changing the world; real-time event streaming is changing the world. In this talk we explore some of best practices to synergistically combine the power of these paradigm shifts to achieve a much greater return on your Kafka investments. From declarative deployments, zero-downtime upgrades, elastic scaling to self-healing and automated governance, learn how you can bring the next level of speed, agility, resilience, and security to your Kafka implementations.

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | J...

HostedbyConfluent

As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Apache Kafka. We believe that CIP positions us for the best defense against cyber threats well into the future. Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It achieves economies of scale by acquiring data once and consuming it many times. It reduces technical debt by eliminating custom point-to-point connections for producing and consuming data. At the same time, it provides the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately lead to better prevention. In our session, we’ll discuss the details described in the IT@Intel white paper that was published in Nov 2020 with same title. We’ll share some stream processing techniques, such as filtering and enriching in Kafka to deliver contextually rich data to Splunk and many of our security controls.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions. Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

HostedbyConfluent

When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.

Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix

HostedbyConfluent

The business systems of an organization are a continuous source of events. Each system also needs to know about events happening in the other systems. Exchanging these events through direct API calls creates a web of inter-dependencies, is fragile and fails to scale. We examine how this problem can be solved through the use of right integration patterns implemented as a light-weight event hub that leverages the power of Kafka and Confluent to operate at enterprise scale. We demonstrate how JavaScript with its event-driven programming model can be a good fit for implementing an event hub that ensures guaranteed message delivery in the face of failures within the individual subscriber systems. Many organizations having large engineering teams skilled in NodeJS and a multitude of NodeJs applications. We show how these teams can easily leverage the power of Kafka and scale their applications with the right architectural building blocks. We also offer insights from our own experience of building NodeJS based Kafka applications.

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...

HostedbyConfluent

SIEM platforms are essential to the new cybersecurity paradigm and data collection layer is a very important piece of it. When you deliver a new platform, you can easily get lost in a variety of different vendors and solutions, too many challenges are facing. What if I change vendors, will I keep my data? How to feed multiple tools with the same data? How to collect data from custom apps and services? How to pay less for an expensive platform? How to keep data without a huge cost? Join us if you are looking for the answers. In this session, you will learn how we replaced the vendor-provided data collection layer with kafka connect and the lessons we learnt. After the talk you will know: - architecture and real-life examples of the flexible and highly available data collection platform - custom connectors that do most of the work for us and how to extend the connectors to consume new data, we made them open sourced - easy way to receive data from thousands of servers and many cloud services - how to archive data at low cost You will leave armed with a set of free tools and recipes to build a truly vendor-agnostic data collection platform. It will allow you to take you SIEM costs under control. You will feed your analytics tools with what they need and archive the rest at low cost. You will feed your SIEM smart!

How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Ros...

HostedbyConfluent

Apache Kafka fundamentally changes how organizations build and deploy a universal data pipeline that is scalable, reliable, and durable enough to meet the needs of digital-first organizations. However, as powerful as Kafka is today, it’s not an event-streaming platform - and getting it there on your own is a long, complicated, and expensive process. Earlier this year Confluent announced Project Metamorphosis - our plan to bring the best characteristics of cloud native systems to Apache Kafka. Since May we’ve begun transforming Confluent Cloud and Confluent Platform to do just that. Join two of our Product Managers, Dan Rosanova and Addison Huddy to: Learn how we’ve evolved Confluent Cloud with the first phase of Project Metamorphosis releases See how Confluent Platform 6.0 brings these transformational, cloud-like qualities to self-managed Kafka Get a sneak peak of our next Metamorphosis theme and how it impacts your Kafka and event-streaming strategy.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...

HostedbyConfluent

Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services. At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure. This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.

Understanding Kafka Produce and Fetch api calls for high throughtput applicat...

HostedbyConfluent

The data team at Cloudflare uses Kafka to process tens of petabytes a day. All this data is moved using the 2 foundational Kafka api calls: Produce (api key 0) and Fetch (api key 1). Understanding the structure of these calls (and of the underlying RecordSet structure) is key to building high throughput clients. The talk describes the basics of the Kafka wire protocol (api keys, correlation id), and the structure of the Produce and Fetch calls. It shows how the asynchronous nature of the wire protocol can combine with the structure of the Produce and Fetch calls to increase latency and reduce client throughput; a solution is offered through use of synchronous single-partition calls. The RecordSet structure, which is used to encode and store sets (batches) of records is described, and its implications on Fetch requests are discussed. The relationship between Fetch api calls and ""consume"" operations is discussed, as is the impact of offset alignment to RecordSet boundaries.

Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik

HostedbyConfluent

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...

HostedbyConfluent

Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.

VMworld 2013: Virtualizing Highly Available SQL Servers

VMworld

5 Quick Wins for the Cloud

RightScale

RightScale Webinar: So you want to move to the cloud... but you’re not sure what that means, or where you would even start. Or you want to get your feet wet with a proof-of-concept project before you bring out the big guns. We asked Brian Adler, our Professional Services Architect who works directly with customers on cloud projects every single day, to select five cloud projects that you can get started with (and complete!) quickly. In this webinar, Brian and Rafael Saavedra, our VP of Engineering, will walk you through those five projects and will help you demonstrate success in the cloud now.

What's hot

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

HostedbyConfluent

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff ...

HostedbyConfluent

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

HostedbyConfluent

Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...

HostedbyConfluent

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

HostedbyConfluent

Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset

HostedbyConfluent

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

HostedbyConfluent

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | J...

HostedbyConfluent

As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Apache Kafka. We believe that CIP positions us for the best defense against cyber threats well into the future. Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It achieves economies of scale by acquiring data once and consuming it many times. It reduces technical debt by eliminating custom point-to-point connections for producing and consuming data. At the same time, it provides the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately lead to better prevention. In our session, we’ll discuss the details described in the IT@Intel white paper that was published in Nov 2020 with same title. We’ll share some stream processing techniques, such as filtering and enriching in Kafka to deliver contextually rich data to Splunk and many of our security controls.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

HostedbyConfluent

Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix

HostedbyConfluent

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...

HostedbyConfluent

How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Ros...

HostedbyConfluent

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...

HostedbyConfluent

Understanding Kafka Produce and Fetch api calls for high throughtput applicat...

HostedbyConfluent

Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik

HostedbyConfluent

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...

HostedbyConfluent

What's hot (20)