Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

•

2 likes•7,652 views

Let's begin a very unusual Kafka Summit by reflecting about change. Changes we've seen in the software engineering world and changes we've seen in Kafka. We'll also talk about things that don't change - like great software design and architecture. We'll dive deep into two huge changes that are happening in the Kafka community right now - and the possibilities they open for the future.

Technology

The World Changed
Cloud
Kafka
Use cases
JavaRaftHardware

Great Architectures
Have Conceptual Integrity

Picking the right abstraction
opens up the inherent efficiencies
of the hardware itself.

High Performance from First Principles
Optimize
like crazy
Work with the
system, not
against it
Understand
the hardware,
OS, Java

Rethinking Kafka’s
Architecture Principles

Adding brokers will
increase capacity
Multi-tenancy
Linearly, in all
dimensions
Company-wide Scale

Great
documentation
Observable
Easy to
automate
Plays well
with others
Efficient
(= cheap)
Operationally Friendly

Easy to add /
remove brokers
Adapt to changing
workloads
Elasticity

KIP-500
Replace Zookeeper with
Metadata Quorum
Control Plane
KIP-405
Kafka Tiered Storage
Data Plane
Design Considerations in Action

KAFKA COLD STORAGE
Add Kafka Broker
Principles in Action: Elasticity

Infinite
history
Pluggable
storage
Easy capacity
planning
Additional Benefits of Storage Layer

What to do with
all the time you’ll save?

Principles in Action:
Conceptual Integrity
• We love logs
• We want to use logs for
everything

Async communication
via shared log vs.
point-to-point
Materialize metadata
state in memory
10,000,000 partitions
Principles in Action: Scalability

Tuning
GC
Tweaking
servers
Security
Upgrades
Patches
Capacity
Backup

No need to manage
2 distributed systems
History of state changes
in an immutable log
Principles in Action: Operationally Friendly

Elastic InfiniteIntegrated
The New Kafka

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...

HostedbyConfluent

Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security. A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka. In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.

Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...

HostedbyConfluent

When choosing an event streaming platform, Kafka shouldn’t be the only technology you look at. There are a plethora of others in the messaging space today, including open source and proprietary software as well as a range of cloud services. So how do you know you are choosing the right one? A great way to deepen our understanding of event streaming and Kafka is exploring the trade-offs in distributed system design and learning about the choices made by the Kafka project. We’ll look at how Kafka stacks up against other technologies in the space, including traditional messaging systems like Apache ActiveMQ and RabbitMQ as well as more contemporary ones, such as BookKeeper derivatives like Apache Pulsar or Pravega. This talk focuses on the technical details such as difference in messaging models, how data is stored locally as well as across machines in a cluster, when (not) to add tiers to your system, and more. By the end of the talk, you should have a good high-level understanding of how these systems compare and which you should choose for different types of use cases.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...

confluent

The last 5 years, Kafka and Flink have become mature technologies that have allowed us to embrace the streaming paradigm. You can bet on them to build reliable and efficient applications. They are active projects backed by companies using them in production. They have a good community contributing, and sharing experience and knowledge. Kafka and Flink are solid choices if you want to build a data platform that your data scientists or developers can use to collect, process, and distribute data. You can put together Kafka Connect, Kafka, Schema Registry, and Flink. First, you will take care of their deployment. Then, for each case, you will setup each part, and of course develop the Flink job so it can integrate easily with the rest. Looks like a challenging but exciting project, isn't it? In this session, you will learn how you can build such data platform, what are the nitty-gritty of each part, how you can plug them together, in particular how to plug Flink in the Kafka ecosystem, what are the common pitfalls to avoid, and what it requires to be deployed on kubernetes. Even if you are not familiar with all the technologies, there will be enough introduction so you can follow. Come and learn how we can actually cross the streams!

Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...

HostedbyConfluent

Apache Kafka is a key part of the Big Data infrastructure at Salesforce, enabling publish/subscribe and data transport in near real-time at enterprise scale handling trillions of messages per day. In this session, hear from the teams at Salesforce that manage Kafka as a service, running over a hundred clusters across on-premise and public cloud environments with over 99.9% availability. Hear about best practices and innovations, including: * How to manage multi-tenant clusters in a hybrid environment * High volume data pipelines with Mirus replicating data to Kafka and blob storage * Kafka Fault Injection Framework built on Trogdor and Kibosh * Automated recovery without data loss * Using Envoy as an SNI-routing Kafka gateway We hope the audience will have practical takeaways for building, deploying, operating, and managing Kafka at scale in the enterprise.

Utilizing Kafka Connect to Integrate Classic Monoliths into Modern Microservi...

HostedbyConfluent

Having started with classic monolith applications in the late 90s and adopting a new microservice architecture in 2015, our organization needed a convenient, reliable, and low-cost way to push changes back and forth between them. One that preferably utilized technology already on hand and could exchange information between multiple data stores. In this session we will explore how Kafka Connect and its various connectors satisfied this need. We will review the two disparate tech stacks we needed to integrate, and the strategies and connectors we used to exchange information. Finally, we will cover some enhancements we made to our own processes including integrating Kafka Connect and its connectors into our CI/CD pipeline and writing tools to monitor connectors in our production environment.

Real time dashboards with Kafka and Druid

Venu Ryali

Cloud is changing the world; Kubernetes is changing the world; real-time event streaming is changing the world. In this talk we explore some of best practices to synergistically combine the power of these paradigm shifts to achieve a much greater return on your Kafka investments. From declarative deployments, zero-downtime upgrades, elastic scaling to self-healing and automated governance, learn how you can bring the next level of speed, agility, resilience, and security to your Kafka implementations.

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

HostedbyConfluent

As a data professional, you are the glue that makes cross-platform integrations possible. With the increase in adoption of hybrid cloud architectures, Kafka is an increasingly relevant tool for building data pipelines between platforms and accelerating delivery on cloud projects. Early exposure to Kafka on Azure capabilities gives you an edge to build better mousetraps at the design phase. Customers already running Kafka on premises and are looking to extend Kafka systems to Azure can get started quickly with Confluent Cloud. Additionally, DevOps for self-managed options can be easily scalable with Ansible for Virtual Machines or containers via Azure Kubernetes Services or Azure Container Instances. This session is presented from the Microsoft Solution Architect perspective by Israel Ekpo, Microsoft Cloud Solution Architect and Alicia Moniz, Microsoft MVP. They will cover use cases and scenarios, along with key Azure integration points and architecture patterns.

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019

confluent

Namely is a late-stage startup that builds HR, Payroll and Benefits software for mid-sized businesses. Over the years, we've ended up with a number of monolithic and legacy applications covering overlapping domain concepts, which has limited our ability to deliver new and innovative features to our customers. We need a way to get our data out of the monoliths to decouple our systems and increase our velocity. We've chosen Kafka as our way to liberate our data in a reliable, scalable and maintainable way. This talk covers specific examples of successes and missteps in our move to Kafka as the backbone of our architecture. It then looks to the future - where we are trying to go, and how we plan on getting, both from the short term and long term perspectives. Key Takeaways: - Successful and unsuccessful approaches to gradually introducing Kafka to a large organization in a way that meets the short and long term needs of the business. - Successful and unsuccessful patterns for using Kafka. - Pragmaticism versus purisim: Building Kafka-first systems, and migrating legacy systems to Kafka with Debezium. - Combining event driven systems with RPC based systems. Observability, alerting and testing. - Actionable steps that you can take to your organization to help drive adoption.

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

HostedbyConfluent

One of the key metrics to monitor when working with Apache Kafka, as a data pipeline or a streaming platform, is Consumer Groups Lag. Lag is the delta between the last produced message and the last committed message of a partition. In other words, lag indicates how far behind your application is in processing up-to-date information. For a long time, we used our own service to keep track of these metrics, collect them and visualize them. But this didn’t scale well. You had to perform many manual operations, redeploy it and to do other tedious manual tasks, but most importantly, the biggest gap for us, was that its output was represented in absolute numbers (e.g - your lag is 30K), which basically tells you nothing as a human being. We understood that we had to find a more suitable solution that will give us better visibility and will allow us to measure the lag in a time-based format that we all understand. In this talk, I’m going to go over the core concepts of Kafka offsets and lags, and explain why lag even matters and is an important KPI to measure. I’ll also talk about the kind of research we did to find the right tool, what the options in the market were at the time, and eventually why we chose Linkedin’s Burrow as the right tool for us. And finally, I’ll take a closer look at Burrow, its building blocks, how we build and deploy it, how we monitor better with it, and eventually the most important improvement - how we transformed its output from numbers to time-based metrics.

Kafka Summit SF 2017 - Database Streaming at WePay

confluent

Building a derived data store using Kafka

Venu Ryali

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

HostedbyConfluent

You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.

Change Data Capture using Kafka

Akash Vacher

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020

confluent

Kafka is one of the most important foundation services at Zendesk. It became even more crucial with the introduction of Global Event Bus which my team built to propagate events between Kafka clusters hosted at different parts of the world and between different products. As part of its rollout, we had to add mTLS support in all of our Kafka Clusters (we have quite a few of them), this was to make propagation of events between clusters hosted at different parts of the world secure. It was quite a journey, but we eventually built a solution that is working well for us. Things I will be sharing as part of the talk: 1. Establishing the use case/problem we were trying to solve (why we needed mTLS) 2. Building a Certificate Authority with open source tools (with self-signed Root CA) 3. Building helper components to generate certificates automatically and regenerate them before they expire (helps using a shorter TTL (Time To Live) which is good security practice) for both Kafka Clients and Brokers 4. Hot reloading regenerated certificates on Kafka brokers without downtime 5. What we built to rotate the self-signed root CA without downtime as well across the board 6. Monitoring and alerts on TTL of certificates 7. Performance impact of using TLS (along with why TLS affects kafka’s performance) 8. What we are doing to drive adoption of mTLS for existing Kafka clients using PLAINTEXT protocol by making onboarding easier 9. How this will become a base for other features we want, eg ACL, Rate Limiting (by using the principal from the TLS certificate as Identity of clients)

Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR

confluent

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

confluent

Performance Tuning RocksDB for Kafka Streams’ State Stores

confluent

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...

HostedbyConfluent

Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability. This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment: kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability. kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters. We'll also present challenges we encountered along the way and share our learnings with the audience.

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

confluent

Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020

confluent

Apache Kafka sits at the center of a technology ecosystem that can be a bit overwhelming to someone just getting started. Fortunately, Apache Kafka is also at the heart of an amazing community that is able and eager to help! So, if you are new, or relatively new, to Apache Kafka, welcome! I’d like to introduce you to the Kafka ecosystem, and present you with a plan for how to learn and be productive with it. I’d also like to introduce you to one of the most helpful and welcoming software communities I’ve ever encountered. I’ll take you through the basics of Kafka—the brokers, the partitions, the topics—and then on and up into the different APIs and tools that are available to work with it. Consider it a Kafka 101, if you will. We’ll stay at a high level, but we’ll cover a lot of ground, with an emphasis on where and how you can dig in deeper. I am still learning myself, so I will share with you what and who have helped me in my journey, and then I’ll invite you to continue that journey with me. It’s going to be a great adventure!

Should you read Kafka as a stream or in batch? Should you even care? | Ido Na...

HostedbyConfluent

Should you consume Kafka in a stream OR batch? When should you choose each one? What is more efficient, and cost effective? In this talk we’ll give you the tools and metrics to decide which solution you should apply when, and show you a real life example with cost & time comparisons. To highlight the differences, we’ll dive into a project we’ve done, transitioning from reading Kafka in a stream to reading it in batch. By turning conventional thinking on its head and reading our multi-petabyte Kafka stream in batch using Spark and Airflow, we’ve achieved a huge cost reduction of 65% while at the same time getting a more scalable and resilient solution. We’ll explore the tradeoffs and give you the metrics and intuition you’ll need to make such decisions yourself. We’ll cover: Costs of processing in stream compared to batch Scaling up for bursts and reprocessing Making the tradeoff between wait times and costs Recovering from outages And much more…

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

confluent

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...

HostedbyConfluent

Whether you are deploying a new application in Microservices or transitioning from a monolithic database application to a cloud-ready architecture, you will inevitably face the decision of either creating a service mesh of API’s – or – using an event bus for better durability, reliability and extensibility of your application. If you choose to go the event bus route, Kafka is an excellent choice for several reasons. One key technology not to overlook is Avro Schemas. They provide a definition for your event payload, just like an API, to ensure all of the event consumers can reliably consume the events. They also handle schema evolution as requirements change and much, much more. In this talk we will discuss all the nuances and considerations around using Avro Schemas for your JSON event payloads. From developer tools, to DevOps approaches, versioning, governance and some “gotchas” we found when working with Avro Schemas and the Confluent Schema Registry.

Deploying Kafka Streams Applications with Docker and Kubernetes

confluent

(Gwen Shapira + Matthias J. Sax, Confluent) Kafka Summit SF 2018 Kafka Streams, Apache Kafka’s stream processing library, allows developers to build sophisticated stateful stream processing applications which you can deploy in an environment of your choice. Kafka Streams is not only scalable, but fully elastic allowing for dynamic scale-in and scale-out as the library handles state migration transparently in the background. By running Kafka Streams applications on Kubernetes, you will be able to use Kubernetes powerful control plane to standardize and simplify the application management—from deployment to dynamic scaling. In this technical deep dive, we’ll explain the internals of dynamic scaling and state migration in Kafka Streams. We’ll then show, with a live demo, how a Kafka Streams application can run in a Docker container on Kubernetes and the dynamic scaling of an application running in Kubernetes.

Deploying Confluent Platform for Production

confluent

DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf

confluent

Reinventing Kafka in the Data Streaming Era - Jun Rao

confluent

What's hot

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

HostedbyConfluent

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

HostedbyConfluent

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019

confluent

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

HostedbyConfluent

Kafka Summit SF 2017 - Database Streaming at WePay

confluent

Building a derived data store using Kafka

Venu Ryali

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

HostedbyConfluent

Change Data Capture using Kafka

Akash Vacher

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020

confluent

Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR

confluent

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

confluent

Performance Tuning RocksDB for Kafka Streams’ State Stores

confluent

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...

HostedbyConfluent

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

confluent

Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020

confluent

Should you read Kafka as a stream or in batch? Should you even care? | Ido Na...

HostedbyConfluent

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

confluent

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...

HostedbyConfluent

Deploying Kafka Streams Applications with Docker and Kubernetes

confluent

Deploying Confluent Platform for Production

confluent

What's hot (20)

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

Confluent On Azure: Why you should add Confluent to your Azure toolkit | Alic...

Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019

Kafka Lag Monitoring For Human Beings (Elad Leev, AppsFlyer) Kafka Summit 2020

Kafka Summit SF 2017 - Database Streaming at WePay

Building a derived data store using Kafka

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

Change Data Capture using Kafka

Securing Kafka At Zendesk (Joy Nag, Zendesk) Kafka Summit 2020

Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR

Kafka Summit NYC 2017 - Apache Kafka in the Enterprise: What if it Fails?

Performance Tuning RocksDB for Kafka Streams’ State Stores

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...

Kafka Summit SF 2017 - Fast Data in Supply Chain Planning

Welcome to Kafka; We’re Glad You’re Here (Dave Klein, Centene) Kafka Summit 2020

Should you read Kafka as a stream or in batch? Should you even care? | Ido Na...

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregation

Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...

Deploying Kafka Streams Applications with Docker and Kubernetes

Deploying Confluent Platform for Production

Similar to Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf

confluent

Reinventing Kafka in the Data Streaming Era - Jun Rao

confluent

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

HostedbyConfluent

When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.

Stratoscale Latest and GreatestZach Lanksbury

Gestione gerarchica dei dati con SUSE Enterprise Storage e HPE DMF

SUSE Italy

Red Hat Storage Roadmap

Red_Hat_Storage

Red Hat Storage Roadmap

Colleen Corrice

2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructure

devopsdaysaustin

OpenStack Preso: DevOps on Hybrid Infrastructure

rhirschfeld

Using next gen storage in Cloudstack

ShapeBlue

Relay: The Next Leg, Eric Sorenson, Puppet

Puppet

Revolutionary Storage for Modern Databases, Applications and Infrastrcture

sabnees

VMworld 2013: Architecting Oracle Databases on vSphere 5 with NetApp Storage

VMworld

New Features in Confluent Platform 6.0 / Apache Kafka 2.6

Kai Wähner

SAP Workloads on AWS

Amazon Web Services

OCI Overview

Kamil Wieczorek

6Reinventing Oracle Systems in a Cloudy World (RMOUG Trainingdays, February 2...

Lucas Jellema

The cloud is changing many things. Even the decision to not (yet) adopt cloud is one to make explicitly. Now is a time for any organization to reconsider the IT landscape. For each system we should make a conscious ruling on its roadmap. The 6R model suggests six ways to move a system forward. This session uses the 6R model and applies it specifically to Oracle technology based systems: what are the options and considerations for Oracle Database, Oracle Fusion Middleware, custom applications, and other red components? What future should we consider and how do we choose? The paths chosen by several Oracle-heavy users is presented to illustrate these options and the decision making process. Oracle Cloud Infrastructure and Autonomous Database play a role, as do Azure IaaS and Azure Managed Database as well as on premises systems. Latency, recovery, scalability, licenses, automation, lock-in, skills, and resources all make their appearance.

Red Hat Ceph Storage: Past, Present and Future

Red_Hat_Storage

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Alluxio, Inc.

Alluxio Tech Talk January 21, 2020 Speakers: Matt Fuller, Starburst Dipti Borkar, Alluxio With the advent of the public clouds and data increasingly siloed across many locations -- on premises and in the public cloud -- enterprises are looking for more flexibility and higher performance approaches to analyze their structured data. Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn more about: - The architecture of Presto, an open source distributed SQL engine - How the Presto + Alluxio stack queries data from cloud object storage like S3 for faster and more cost-effective analytics - Achieving data locality and cross-job caching with Alluxio regardless of where data is persisted

Ai big dataconference_jeffrey ricker_kappa_architecture

Olga Zinkevych

Topic of presentation: Kappa architecture (and beyond) The main points of the presentation: We will discuss the evolution of big data architecture, from batch to Lambda to Kappa. I will walk through how to implement a Kappa architecture with practical examples, focusing on how to reach full potential and avoid the pitfalls. We will finish with reviewing what lies ahead, including the inevitable consolidation between microservices, GPGPU and Hadoop. http://dataconf.com.ua/index.php#agenda #dataconf #AIBDConference

Similar to Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture (20)

DIMT 2023 SG - Hands-on Workshop_ Getting started with Confluent Cloud.pdf

Reinventing Kafka in the Data Streaming Era - Jun Rao

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

Stratoscale Latest and Greatest

Gestione gerarchica dei dati con SUSE Enterprise Storage e HPE DMF

Red Hat Storage Roadmap

2016 - Open Mic - IGNITE - Open Infrastructure = ANY Infrastructure

OpenStack Preso: DevOps on Hybrid Infrastructure

Using next gen storage in Cloudstack

Relay: The Next Leg, Eric Sorenson, Puppet

Revolutionary Storage for Modern Databases, Applications and Infrastrcture

VMworld 2013: Architecting Oracle Databases on vSphere 5 with NetApp Storage

New Features in Confluent Platform 6.0 / Apache Kafka 2.6

SAP Workloads on AWS

OCI Overview

6Reinventing Oracle Systems in a Cloudy World (RMOUG Trainingdays, February 2...

Red Hat Ceph Storage: Past, Present and Future

Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack

Ai big dataconference_jeffrey ricker_kappa_architecture

More from confluent

Speed Wins: From Kafka to APIs in Minutes

confluent

Evolving Data Governance for the Real-time Streaming and AI Era

confluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

confluent

Santander Stream Processing with Apache Flink

confluent

Unlocking the Power of IoT: A comprehensive approach to real-time insights

confluent

Workshop híbrido: Stream Processing con Flink

confluent

El Stream processing es un requisito previo de la pila de data streaming, que impulsa aplicaciones y pipelines en tiempo real. Permite una mayor portabilidad de datos, una utilización optimizada de recursos y una mejor experiencia del cliente al procesar flujos de datos en tiempo real. En nuestro taller práctico híbrido, aprenderás cómo filtrar, unir y enriquecer fácilmente datos en tiempo real dentro de Confluent Cloud utilizando nuestro servicio Flink sin servidor.

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

confluent

Our talk will explore the transformative impact of integrating Confluent, HiveMQ, and SparkPlug in Industry 4.0, emphasizing the creation of a Unified Namespace. In addition to the creation of a Unified Namespace, our webinar will also delve into Stream Governance and Scaling, highlighting how these aspects are crucial for managing complex data flows and ensuring robust, scalable IIoT-Platforms. You will learn how to ensure data accuracy and reliability, expand your data processing capabilities, and optimize your data management processes. Don't miss out on this opportunity to learn from industry experts and take your business to the next level.

AWS Immersion Day Mapfre - Confluent

confluent

La arquitectura impulsada por eventos (EDA) será el corazón del ecosistema de MAPFRE. Para seguir siendo competitivas, las empresas de hoy dependen cada vez más del análisis de datos en tiempo real, lo que les permite obtener información y tiempos de respuesta más rápidos. Los negocios con datos en tiempo real consisten en tomar conciencia de la situación, detectar y responder a lo que está sucediendo en el mundo ahora.

Eventos y Microservicios - Santander TechTalk

confluent

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

confluent

Citi TechTalk Session 2: Kafka Deep Dive

confluent

Build real-time streaming data pipelines to AWS with Confluent

confluent

Traditional data pipelines often face scalability issues and challenges related to cost, their monolithic design, and reliance on batch data processing. They also typically operate under the premise that all data needs to be stored in a single centralized data source before it's put to practical use. Confluent Cloud on Amazon Web Services (AWS) provides a fully managed cloud-native platform that helps you simplify the way you build real-time data flows using streaming data pipelines and Apache Kafka.

Q&A with Confluent Professional Services: Confluent Service Mesh

confluent

Citi Tech Talk: Event Driven Kafka Microservices

confluent

Confluent & GSI Webinars series - Session 3

confluent

An in depth look at how Confluent is being used in the financial services industry. Gain an understanding of how organisations are utilising data in motion to solve common problems and gain benefits from their real time data capabilities. It will look more deeply into some specific use cases and show how Confluent technology is used to manage costs and mitigate risks. This session is aimed at Solutions Architects, Sales Engineers and Pre Sales, and also the more technically minded business aligned people. Whilst this is not a deeply technical session, a level of knowledge around Kafka would be helpful.

Citi Tech Talk: Messaging Modernization

confluent

Transforming applications built with traditional messaging solutions such as TIBCO, MQ and Solace to be scalable, reliable and ready for the move to cloud How can applications built with traditional messaging technologies like TIBCO, Solace and IBM MQ be modernised and be made cloud ready? What are the advantages to Event Streaming approaches to pub/sub vs traditional message queues? What are the strengeths and weaknesses of both approaches, and what use cases and requirements are actually a better fit for messaging than Kafka?

Citi Tech Talk: Data Governance for streaming and real time data

confluent

Confluent & GSI Webinars series: Session 2

confluent

Data In Motion Paris 2023

confluent

Vous apprendrez également à : • Créer plus rapidement des produits et fonctionnalités à l’aide d’une suite complète de connecteurs et d’outils de gestion des flux, et à connecter vos environnements à des pipelines de données • Protéger vos données et charges de travail les plus critiques grâce à des garanties intégrées en matière de sécurité, de gouvernance et de résilience • Déployer Kafka à grande échelle en quelques minutes tout en réduisant les coûts et la charge opérationnelle associés

Confluent Partner Tech Talk with Synthesis

confluent

More from confluent (20)

Speed Wins: From Kafka to APIs in Minutes

Evolving Data Governance for the Real-time Streaming and AI Era

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...

Santander Stream Processing with Apache Flink

Unlocking the Power of IoT: A comprehensive approach to real-time insights

Workshop híbrido: Stream Processing con Flink

Industry 4.0: Building the Unified Namespace with Confluent, HiveMQ and Spark...

AWS Immersion Day Mapfre - Confluent

Eventos y Microservicios - Santander TechTalk

Q&A with Confluent Experts: Navigating Networking in Confluent Cloud

Citi TechTalk Session 2: Kafka Deep Dive

Build real-time streaming data pipelines to AWS with Confluent

Q&A with Confluent Professional Services: Confluent Service Mesh

Citi Tech Talk: Event Driven Kafka Microservices

Confluent & GSI Webinars series - Session 3

Citi Tech Talk: Messaging Modernization

Citi Tech Talk: Data Governance for streaming and real time data

Confluent & GSI Webinars series: Session 2

Data In Motion Paris 2023

Confluent Partner Tech Talk with Synthesis

Recently uploaded

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

Knowledge engineering: from people to machines and back

Elena Simperl

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

Generating a custom Ruby SDK for your web service or Rails API using Smithy

g2nightmarescribd

Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.

The Future of Platform Engineering

Jemma Hussein Allen

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

Jeffrey Haguewood

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams. Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Product School

Recently uploaded (20)

Monitoring Java Application Security with JDK Tools and JFR Events

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Epistemic Interaction - tuning interfaces to provide information for AI support

Knowledge engineering: from people to machines and back

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

When stars align: studies in data quality, knowledge graphs, and machine lear...

Generating a custom Ruby SDK for your web service or Rails API using Smithy

The Future of Platform Engineering

Accelerate your Kubernetes clusters with Varnish Caching

Leading Change strategies and insights for effective change management pdf 1.pdf

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

PCI PIN Basics Webinar from the Controlcase Team

GraphRAG is All You need? LLM & Knowledge Graph

How world-class product teams are winning in the AI era by CEO and Founder, P...

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...

Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture

1. APAC H E KAF KA THE NEXT 10 YEARS

2. The World Changed Cloud Kafka Use cases JavaRaftHardware

3. Evolution of Kafka

4. Great Architectures Have Conceptual Integrity

5. Our North Star

6. Picking the right abstraction opens up the inherent efficiencies of the hardware itself.

7. High Performance from First Principles Optimize like crazy Work with the system, not against it Understand the hardware, OS, Java

8. Rethinking Kafka’s Architecture Principles

9. Adding brokers will increase capacity Multi-tenancy Linearly, in all dimensions Company-wide Scale

10. Great documentation Observable Easy to automate Plays well with others Efficient (= cheap) Operationally Friendly

11. Easy to add / remove brokers Adapt to changing workloads Elasticity

12. KIP-500 Replace Zookeeper with Metadata Quorum Control Plane KIP-405 Kafka Tiered Storage Data Plane Design Considerations in Action

13. KIP-405 Tiered Storage

14. KAFKA COLD STORAGE Add Kafka Broker Principles in Action: Elasticity

15. Infinite history Pluggable storage Easy capacity planning Additional Benefits of Storage Layer

16. What to do with all the time you’ll save?

17. ZooKeeper Removal + New Controller

18. Principles in Action: Conceptual Integrity • We love logs • We want to use logs for everything

19. Async communication via shared log vs. point-to-point Materialize metadata state in memory 10,000,000 partitions Principles in Action: Scalability

20. Tuning GC Tweaking servers Security Upgrades Patches Capacity Backup

21. No need to manage 2 distributed systems History of state changes in an immutable log Principles in Action: Operationally Friendly

22. Next-gen Kafka Architecture

23. Elastic InfiniteIntegrated The New Kafka

24. Our Agile Future