Restate: Stream Processing, but for Microservices
•
0 likes•182 views
"Developing applications in a microservice architecture (MSA) is hard. For all the benefits that MSAs bring over monolithic application architectures, they expose developers to all sorts of tough distributed systems problems, making it non-trivial to build applications that are consistent, scalable, and resilient. Building applications in an event-driven way is a great start to get MSA complexity under control, by leveraging the guarantees of logs/message queues (like Kafka) to offload some of the problems. Stream processing systems (KStreams, Flink, etc.) can be powerful tools as well. Their tight integration of state, messaging, and application logic yields strong guarantees, like exactly-once state & event-delivery semantics. However, having been built originally for data processing, those systems cannot easily express business logic with complex control-flow, orchestration tasks, or workflows. Existing systems also have very different operational characteristics compared to stateless service deployments. In this session, we share ideas from a novel system we are developing, called 'Restate'. Our work is inspired by event-sourcing and stream processing systems, but rethought from the ground up for microservices. Restate connects services with each other (and between services and Kafka), and handles crucial aspects like state, communication, failover, and consistency. It does that with a programming model that is crafted not for data processing or analytics, but for services and applications with complex interactions and control flow. The result is an elegant experience of writing applications that are simple, consistent, yet highly resilient and scalable. We believe that Restate complements the current landscape of event-processing systems, by offering a solution for cases where events facilitate control flow and coordination, rather than real-time data processing."
Report
Share
Report
Share
Download to read offline
Recommended
Restate: Event-driven Asynchronous Services, Easy as Synchronous RPC
"Event-driven services are a popular pattern in microservice architectures. Compared to RPC services, they are typically more resilient (especially when using durable logs, like Kafka), efficient, and scalable. Plus they lend themselves to advanced techniques like bootstrapping/reinstating the service from an existing event log.
But writing event-driven services is much harder than writing RPC-based services: Instead of writing simple request/reply handlers, developers must express logic as asynchronous event-handling logic, introduce complex state machines (and handle concerns of their consistency and persistence). Logic gets spread logic across different event handlers. Applications need to be aware of subscriptions, hot partitions, slow re-balancing, head-of-the-line waiting issues, etc. Many companies that adopt event-driven services find themselves building extensive tools to support dead letter queues, delayed events, and handle retries/recovery/idempotency.
We are developing Restate, to make event-driven applications as easy as RPC-based application. Developers write RPC-based services and Restate adds lightweight durable execution to the handlers (making them suspendable and allowing them to recover partial progress), persistent communication, and request ordering. The result is that code written as simple request/response RPCs executes with the same resilience and efficiency as an async state machine driven by events from a persistent queue, but remains easy to write, reason about, debug, and observe.
Restate natively integrates with Kafka, allowing users to directly process events of Kafka topics with Restate durable async/await programming model. We believe that Restate complements the current landscape of event-processing systems, by radically simplifying transactional event-driven applications that require complex control flow or interact with many external systems."
Reactive Systems that focus on High Availability with Lerna
This document summarizes a presentation about the Lerna reactive system software stack for building highly available systems. Lerna provides libraries, code samples, and documentation to help developers meet non-functional requirements like high availability and throughput. It was shown that a payment processing system built with Lerna recovers from failures across load balancers, applications, databases and data centers within 10 seconds, achieving 99.9999% availability based on the calculated mean time to repair. The key aspects of Lerna that help meet these goals are its elastic and responsive architecture that can scale to handle 1,000 transactions per second while responding within 100ms under high load.
High-speed, Reactive Microservices 2017
Covers how we built a set of high-speed reactive microservices and maximized cloud/hardware costs while meeting objectives in resilience and scalability. Talks about Akka, Kafka, QBit, in-memory computing, from a practitioners point of view. Based on the talks delivered by Geoff Chandler, Jason Daniel, and Rick Hightower at JavaOne 2016 and SF Fintech at Scale 2017, but updated.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.
Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.
This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Kafka is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore:
• Why cloud-native platforms and why run Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Running Kafka as a Streaming Platform on Container Orchestration
101 mistakes FINN.no has made with Kafka (Baksida meetup)
1. The document summarizes 101 mistakes that Finn.no has made with Kafka. It discusses various configuration mistakes and operational mistakes made when initially adopting and using Kafka, and the consequences of those mistakes.
2. Common mistakes included not considering backwards compatibility of Kafka versions, treating Kafka like a database, not properly defining schemas, and not understanding client-side rebalancing.
3. Finn.no's solutions to address the mistakes included running multiple Kafka clusters during a migration, using fewer Kafka partitions as a default, and adopting better configuration practices tailored to their use cases.
Stephan Ewen - Experiences running Flink at Very Large Scale
This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters
Docebo: history of a journey from legacy to serverless
This document discusses the evolution of application architectures from monolith to microservices to serverless. It describes the benefits and drawbacks of each approach. For monoliths, the benefits are ease of development but downsides include poor scalability and reliability. Microservices improve scalability but introduce more complexity. Serverless architectures allow developers to focus on code while avoiding server management overhead, though cold starts can cause latency issues. The document advocates for using serverless as a "stairway" to focus on rapid iteration, frequent delivery, and creating business value.
Recommended
Restate: Event-driven Asynchronous Services, Easy as Synchronous RPC
"Event-driven services are a popular pattern in microservice architectures. Compared to RPC services, they are typically more resilient (especially when using durable logs, like Kafka), efficient, and scalable. Plus they lend themselves to advanced techniques like bootstrapping/reinstating the service from an existing event log.
But writing event-driven services is much harder than writing RPC-based services: Instead of writing simple request/reply handlers, developers must express logic as asynchronous event-handling logic, introduce complex state machines (and handle concerns of their consistency and persistence). Logic gets spread logic across different event handlers. Applications need to be aware of subscriptions, hot partitions, slow re-balancing, head-of-the-line waiting issues, etc. Many companies that adopt event-driven services find themselves building extensive tools to support dead letter queues, delayed events, and handle retries/recovery/idempotency.
We are developing Restate, to make event-driven applications as easy as RPC-based application. Developers write RPC-based services and Restate adds lightweight durable execution to the handlers (making them suspendable and allowing them to recover partial progress), persistent communication, and request ordering. The result is that code written as simple request/response RPCs executes with the same resilience and efficiency as an async state machine driven by events from a persistent queue, but remains easy to write, reason about, debug, and observe.
Restate natively integrates with Kafka, allowing users to directly process events of Kafka topics with Restate durable async/await programming model. We believe that Restate complements the current landscape of event-processing systems, by radically simplifying transactional event-driven applications that require complex control flow or interact with many external systems."
Reactive Systems that focus on High Availability with Lerna
This document summarizes a presentation about the Lerna reactive system software stack for building highly available systems. Lerna provides libraries, code samples, and documentation to help developers meet non-functional requirements like high availability and throughput. It was shown that a payment processing system built with Lerna recovers from failures across load balancers, applications, databases and data centers within 10 seconds, achieving 99.9999% availability based on the calculated mean time to repair. The key aspects of Lerna that help meet these goals are its elastic and responsive architecture that can scale to handle 1,000 transactions per second while responding within 100ms under high load.
High-speed, Reactive Microservices 2017
Covers how we built a set of high-speed reactive microservices and maximized cloud/hardware costs while meeting objectives in resilience and scalability. Talks about Akka, Kafka, QBit, in-memory computing, from a practitioners point of view. Based on the talks delivered by Geoff Chandler, Jason Daniel, and Rick Hightower at JavaOne 2016 and SF Fintech at Scale 2017, but updated.
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with A...
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Pivotal Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value.
Apache Kafka® is providing developers a critically important component as they build and modernize applications to cloud-native architecture.
This talk will explore:
• Why cloud-native platforms and why run Apache Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Demo: Running Apache Kafka as a Streaming Platform on Kubernetes
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Kafka is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore:
• Why cloud-native platforms and why run Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Running Kafka as a Streaming Platform on Container Orchestration
101 mistakes FINN.no has made with Kafka (Baksida meetup)
1. The document summarizes 101 mistakes that Finn.no has made with Kafka. It discusses various configuration mistakes and operational mistakes made when initially adopting and using Kafka, and the consequences of those mistakes.
2. Common mistakes included not considering backwards compatibility of Kafka versions, treating Kafka like a database, not properly defining schemas, and not understanding client-side rebalancing.
3. Finn.no's solutions to address the mistakes included running multiple Kafka clusters during a migration, using fewer Kafka partitions as a default, and adopting better configuration practices tailored to their use cases.
Stephan Ewen - Experiences running Flink at Very Large Scale
This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters
Docebo: history of a journey from legacy to serverless
This document discusses the evolution of application architectures from monolith to microservices to serverless. It describes the benefits and drawbacks of each approach. For monoliths, the benefits are ease of development but downsides include poor scalability and reliability. Microservices improve scalability but introduce more complexity. Serverless architectures allow developers to focus on code while avoiding server management overhead, though cold starts can cause latency issues. The document advocates for using serverless as a "stairway" to focus on rapid iteration, frequent delivery, and creating business value.
Designing distributed systems
This document summarizes a talk on designing distributed systems. It discusses distributed computing paradigms like microservices and cloud computing. It also covers reactive system principles from the Reactive Manifesto like being responsive, resilient and message-driven. It then explains the actor model used in frameworks like Akka, where actors communicate asynchronously through message passing. It provides examples of using actors for concurrency and fault tolerance.
Confluent Messaging Modernization Forum
Confluent provides a platform for modernizing enterprise messaging infrastructure by leveraging Kafka. Kafka uses an immutable log to share data across producers and consumers in a scalable, fault-tolerant, and efficient manner. This allows enterprises to build real-time applications and enable data-in-motion across the organization. Confluent offers tools like Schema Registry, ksqlDB, and connectors to help standardize data, build stream processing applications, and integrate Kafka with other systems.
Event Streaming Architectures with Confluent and ScyllaDB
Jeff Bean will lead a discussion of event-driven architectures, Apache Kafka, Kafka Connect, KSQL and Confluent Cloud. Then we'll talk about some uses of Confluent and Scylla together, including a co-deployment with Lookout, ScyllaDB and Confluent in the IoT space, and the upcoming native connector.
Event Driven Architecture
This document provides an overview of event-driven architecture and distributed transactions with microservices. It discusses event-driven programming and what constitutes a distributed transaction when using microservices. It then covers some common solutions for distributed transactions like the 2PC and Saga patterns. The Saga pattern uses a choreography or orchestration approach and events to coordinate transactions across services. Message brokers like Kafka, RabbitMQ and Azure Event Hub are discussed for facilitating event-driven architectures. Vert.x and Spring WebFlux are presented as reactive frameworks that are non-blocking and scale well.
Twelve Factor App
Twelve Factor App for Distributed Software Development with Cloud Native Computing using Open Source Technology
Loadbalancers: The fabric for your micro services
The document discusses the importance of load balancers for microservices and container-based architectures. It covers how load balancing needs have evolved from traditional monolithic applications to modern approaches that support dynamic microservices topologies, zero-downtime deployments, client-side load balancing, and advanced patterns like circuit breakers. It also introduces NetScaler CPX as a cloud-native load balancer solution for container environments.
REEF: Towards a Big Data Stdlib
This document discusses resource managers and the REEF (Retainable Evaluation Execution Framework) system. It notes that resource managers allow for true multi-tenancy with many workloads and users, but are typically only suitable for sophisticated applications. It then provides examples of machine learning, graph processing, and SQL/MapReduce workloads. The document outlines some of the challenges of using software "silos" and how REEF aims to avoid silos by allowing different computation models to be composed together using a common resource manager and distributed file system (DFS). It provides high-level overviews of the REEF control flow and data plane architectures.
You need Event Mesh, not Service Mesh - Chris Suszynski [WJUG 301]
You've probably heard about building microservices-style applications, right? It's likely that you've heard that a service mesh (such as Istio) can help you achieve this. Unfortunately, in most cases that's an antipattern. Instead, what you need is the Event Mesh. Using the Event Mesh could help you architect your application into a distributed CQRS-style solution that would eventually reconcile system state.
In this session, you'll learn why you should avoid using blocking API calls when building your microservices, and instead use the CQRS architecture to separate commands and queries. Your architecture for commands should be implemented with asynchronous events, which are processed whenever possible. We'll take some inspiration from the Kubernetes architecture, and how you can model such a reconciliation loop within your own enterprise microservices. All this on top of the Knative framework, as an excellent example of event mesh implementation.
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like:
- What technical constraints of Kubernetes can be obstacles for applications and how to tackle these?
- How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security?
- How to industrialize and govern the migration process?
- How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...
CloudNativeCon North America 2017, Austin (Texas, USA): Talk by Josef Adersberger (@adersberger, CTO at QAware)
Abstract:
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like:
- What technical constraints of Kubernetes can be obstacles for applications and how to tackle these?
- How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security?
- How to industrialize and govern the migration process?
- How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?
HBaseCon2017 Highly-Available HBase
In order to effectively predict and prevent online fraud in real time, Sift Science stores hundreds of terabytes of data in HBase—and needs it to be always available. This talk will cover how we used circuit-breaking, cluster failover, monitoring, and automated recovery procedures to improve our HBase uptime from 99.7% to 99.99% on top of unreliable cloud hardware and networks.
Application Modernisation through Event-Driven Microservices
Microservices have emerged as a widely discussed and adopted way to build modern and scalable applications. They are easier to build, manage and maintain than monoliths due to smaller code bases; they isolate complexity, allowing for smaller, more agile teams to create services, and they are flexible—allowing the use of various platforms, programming languages, and tools since these choices affect only an individual service and a small team at a time.
Architecture of Falcon, a new chat messaging backend system build on Scala
The document describes the architecture of Falcon, a new Scala-based backend chat messaging system. The key aspects of the architecture are:
- It uses an event-driven architecture with CQRS and event sourcing principles. Writing and reading are separate with different data models and storage.
- Events are stored in Kafka for durability and consumed asynchronously by downstream systems to update read models and notify legacy systems.
- The architecture is designed for scalability, high performance, resiliency and low cost through principles like auto-sharding and parallel processing in Kafka.
Flink forward-2017-netflix keystones-paas
The need for gleaning answers from unbounded data streams is moving from nicety to a necessity. Netflix is a data driven company, and has a need to process over 1 trillion events a day amounting to 3 PB of data to derive business insights.
To ease extracting insight, we are building a self-serve, scalable, fault-tolerant, multi-tenant "Stream Processing as a Service" platform so the user can focus on data analysis. I'll share our experience using Flink to help build the platform.
Big Data Streams Architectures. Why? What? How?
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
FreeSWITCH as a Microservice
This document discusses microservices architecture compared to a monolithic architecture. A microservices architecture breaks an application into smaller, independent services that each perform discrete functions. This allows for more rapid development and improved scalability. However, a microservices architecture is also more complex to deploy and manage. The document provides an example of how a VoIP application could use a microservices approach by breaking components like billing, fraud detection, and call analytics into separate services. It also discusses using Docker containers and services to deploy and scale the microservices architecture.
Software architecture for data applications
The document provides an overview of software architecture considerations for data applications. It discusses sample data system components like Memcached, Redis, Elasticsearch, and Solr. It covers topics such as service level objectives, data models, query languages, graph models, data warehousing, machine learning pipelines, and distributed systems. Specific frameworks and technologies mentioned include Spark, Kafka, Neo4j, PostgreSQL, and ZooKeeper. The document aims to help understand architectural tradeoffs and guide the design of scalable, performant, and robust data systems.
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Open Source Summit 2018, Vancouver (Canada): Talk by Josef Adersberger (@adersberger, CTO at QAware), Michael Frank (Software Architect at QAware) and Robert Bichler (IT Project Manager at Allianz Germany)
Abstract:
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud-native apps. But what to do if you’ve no shiny new cloud-native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.
Fundamentals Of Transaction Systems - Part 2: Certainty suppresses Uncertaint...
The document discusses transaction systems and consistency models. It summarizes that:
- Brewer's CAP theorem states that distributed systems can only achieve two of consistency, availability, and partition tolerance.
- Many financial systems achieve all three by using private networks and 3-phase commit, challenging assumptions of the CAP theorem.
- Workflow systems can help achieve consistency across inconsistent distributed systems by driving them into acceptable states.
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way.
Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks.
Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case.
Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."
Renaming a Kafka Topic | Kafka Summit London
"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics.
Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"
More Related Content
Similar to Restate: Stream Processing, but for Microservices
Designing distributed systems
This document summarizes a talk on designing distributed systems. It discusses distributed computing paradigms like microservices and cloud computing. It also covers reactive system principles from the Reactive Manifesto like being responsive, resilient and message-driven. It then explains the actor model used in frameworks like Akka, where actors communicate asynchronously through message passing. It provides examples of using actors for concurrency and fault tolerance.
Confluent Messaging Modernization Forum
Confluent provides a platform for modernizing enterprise messaging infrastructure by leveraging Kafka. Kafka uses an immutable log to share data across producers and consumers in a scalable, fault-tolerant, and efficient manner. This allows enterprises to build real-time applications and enable data-in-motion across the organization. Confluent offers tools like Schema Registry, ksqlDB, and connectors to help standardize data, build stream processing applications, and integrate Kafka with other systems.
Event Streaming Architectures with Confluent and ScyllaDB
Jeff Bean will lead a discussion of event-driven architectures, Apache Kafka, Kafka Connect, KSQL and Confluent Cloud. Then we'll talk about some uses of Confluent and Scylla together, including a co-deployment with Lookout, ScyllaDB and Confluent in the IoT space, and the upcoming native connector.
Event Driven Architecture
This document provides an overview of event-driven architecture and distributed transactions with microservices. It discusses event-driven programming and what constitutes a distributed transaction when using microservices. It then covers some common solutions for distributed transactions like the 2PC and Saga patterns. The Saga pattern uses a choreography or orchestration approach and events to coordinate transactions across services. Message brokers like Kafka, RabbitMQ and Azure Event Hub are discussed for facilitating event-driven architectures. Vert.x and Spring WebFlux are presented as reactive frameworks that are non-blocking and scale well.
Twelve Factor App
Twelve Factor App for Distributed Software Development with Cloud Native Computing using Open Source Technology
Loadbalancers: The fabric for your micro services
The document discusses the importance of load balancers for microservices and container-based architectures. It covers how load balancing needs have evolved from traditional monolithic applications to modern approaches that support dynamic microservices topologies, zero-downtime deployments, client-side load balancing, and advanced patterns like circuit breakers. It also introduces NetScaler CPX as a cloud-native load balancer solution for container environments.
REEF: Towards a Big Data Stdlib
This document discusses resource managers and the REEF (Retainable Evaluation Execution Framework) system. It notes that resource managers allow for true multi-tenancy with many workloads and users, but are typically only suitable for sophisticated applications. It then provides examples of machine learning, graph processing, and SQL/MapReduce workloads. The document outlines some of the challenges of using software "silos" and how REEF aims to avoid silos by allowing different computation models to be composed together using a common resource manager and distributed file system (DFS). It provides high-level overviews of the REEF control flow and data plane architectures.
You need Event Mesh, not Service Mesh - Chris Suszynski [WJUG 301]
You've probably heard about building microservices-style applications, right? It's likely that you've heard that a service mesh (such as Istio) can help you achieve this. Unfortunately, in most cases that's an antipattern. Instead, what you need is the Event Mesh. Using the Event Mesh could help you architect your application into a distributed CQRS-style solution that would eventually reconcile system state.
In this session, you'll learn why you should avoid using blocking API calls when building your microservices, and instead use the CQRS architecture to separate commands and queries. Your architecture for commands should be implemented with asynchronous events, which are processed whenever possible. We'll take some inspiration from the Kubernetes architecture, and how you can model such a reconciliation loop within your own enterprise microservices. All this on top of the Knative framework, as an excellent example of event mesh implementation.
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like:
- What technical constraints of Kubernetes can be obstacles for applications and how to tackle these?
- How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security?
- How to industrialize and govern the migration process?
- How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...
CloudNativeCon North America 2017, Austin (Texas, USA): Talk by Josef Adersberger (@adersberger, CTO at QAware)
Abstract:
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a major German insurance company onto a Kubernetes cluster within one year. We're now close to the finish line and it worked pretty well so far.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way. We'll provide our answers to life, the universe and a cloud native journey like:
- What technical constraints of Kubernetes can be obstacles for applications and how to tackle these?
- How to architect a landscape of hundreds of containerized applications with their surrounding infrastructure like DBs MQs and IAM and heavy requirements on security?
- How to industrialize and govern the migration process?
- How to leverage the possibilities of a cloud native platform like Kubernetes without challenging the tight timeline?
HBaseCon2017 Highly-Available HBase
In order to effectively predict and prevent online fraud in real time, Sift Science stores hundreds of terabytes of data in HBase—and needs it to be always available. This talk will cover how we used circuit-breaking, cluster failover, monitoring, and automated recovery procedures to improve our HBase uptime from 99.7% to 99.99% on top of unreliable cloud hardware and networks.
Application Modernisation through Event-Driven Microservices
Microservices have emerged as a widely discussed and adopted way to build modern and scalable applications. They are easier to build, manage and maintain than monoliths due to smaller code bases; they isolate complexity, allowing for smaller, more agile teams to create services, and they are flexible—allowing the use of various platforms, programming languages, and tools since these choices affect only an individual service and a small team at a time.
Architecture of Falcon, a new chat messaging backend system build on Scala
The document describes the architecture of Falcon, a new Scala-based backend chat messaging system. The key aspects of the architecture are:
- It uses an event-driven architecture with CQRS and event sourcing principles. Writing and reading are separate with different data models and storage.
- Events are stored in Kafka for durability and consumed asynchronously by downstream systems to update read models and notify legacy systems.
- The architecture is designed for scalability, high performance, resiliency and low cost through principles like auto-sharding and parallel processing in Kafka.
Flink forward-2017-netflix keystones-paas
The need for gleaning answers from unbounded data streams is moving from nicety to a necessity. Netflix is a data driven company, and has a need to process over 1 trillion events a day amounting to 3 PB of data to derive business insights.
To ease extracting insight, we are building a self-serve, scalable, fault-tolerant, multi-tenant "Stream Processing as a Service" platform so the user can focus on data analysis. I'll share our experience using Flink to help build the platform.
Big Data Streams Architectures. Why? What? How?
With a current zoo of technologies and different ways of their interaction it's a big challenge to architect a system (or adopt existed one) that will conform to low-latency BigData analysis requirements. Apache Kafka and Kappa Architecture in particular take more and more attention over classic Hadoop-centric technologies stack. New Consumer API put significant boost in this direction. Microservices-based streaming processing and new Kafka Streams tend to be a synergy in BigData world.
FreeSWITCH as a Microservice
This document discusses microservices architecture compared to a monolithic architecture. A microservices architecture breaks an application into smaller, independent services that each perform discrete functions. This allows for more rapid development and improved scalability. However, a microservices architecture is also more complex to deploy and manage. The document provides an example of how a VoIP application could use a microservices approach by breaking components like billing, fraud detection, and call analytics into separate services. It also discusses using Docker containers and services to deploy and scale the microservices architecture.
Software architecture for data applications
The document provides an overview of software architecture considerations for data applications. It discusses sample data system components like Memcached, Redis, Elasticsearch, and Solr. It covers topics such as service level objectives, data models, query languages, graph models, data warehousing, machine learning pipelines, and distributed systems. Specific frameworks and technologies mentioned include Spark, Kafka, Neo4j, PostgreSQL, and ZooKeeper. The document aims to help understand architectural tradeoffs and guide the design of scalable, performant, and robust data systems.
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Open Source Summit 2018, Vancouver (Canada): Talk by Josef Adersberger (@adersberger, CTO at QAware), Michael Frank (Software Architect at QAware) and Robert Bichler (IT Project Manager at Allianz Germany)
Abstract:
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud-native apps. But what to do if you’ve no shiny new cloud-native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Running applications on Kubernetes can provide a lot of benefits: more dev speed, lower ops costs, and a higher elasticity & resiliency in production. Kubernetes is the place to be for cloud native apps. But what to do if you’ve no shiny new cloud native apps but a whole bunch of JEE legacy systems? No chance to leverage the advantages of Kubernetes? Yes you can!
We’re facing the challenge of migrating hundreds of JEE legacy applications of a German blue chip company onto a Kubernetes cluster within one year.
The talk will be about the lessons we've learned - the best practices and pitfalls we've discovered along our way.
Fundamentals Of Transaction Systems - Part 2: Certainty suppresses Uncertaint...
The document discusses transaction systems and consistency models. It summarizes that:
- Brewer's CAP theorem states that distributed systems can only achieve two of consistency, availability, and partition tolerance.
- Many financial systems achieve all three by using private networks and 3-phase commit, challenging assumptions of the CAP theorem.
- Workflow systems can help achieve consistency across inconsistent distributed systems by driving them into acceptable states.
Similar to Restate: Stream Processing, but for Microservices (20)
Event Streaming Architectures with Confluent and ScyllaDB
Event Streaming Architectures with Confluent and ScyllaDB
You need Event Mesh, not Service Mesh - Chris Suszynski [WJUG 301]
You need Event Mesh, not Service Mesh - Chris Suszynski [WJUG 301]
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...
Migrating Hundreds of Legacy Applications to Kubernetes - The Good, the Bad, ...
Application Modernisation through Event-Driven Microservices
Application Modernisation through Event-Driven Microservices
Architecture of Falcon, a new chat messaging backend system build on Scala
Architecture of Falcon, a new chat messaging backend system build on Scala
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Fundamentals Of Transaction Systems - Part 2: Certainty suppresses Uncertaint...
Fundamentals Of Transaction Systems - Part 2: Certainty suppresses Uncertaint...
More from HostedbyConfluent
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way.
Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks.
Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case.
Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."
Renaming a Kafka Topic | Kafka Summit London
"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics.
Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"
Evolution of NRT Data Ingestion Pipeline at Trendyol
"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights.
We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string.
Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring.
The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service.
While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications.
The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection.
Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."
Exactly-once Stream Processing with Arroyo and Kafka
"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output.
Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing.
In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."
Fish Plays Pokemon | Kafka Summit London
"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium.
This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption.
By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."
Tiered Storage 101 | Kafla Summit London
What is tiered storage and what is it good for? After this session you will know how to leverage the tiered storage feature to enable longer retention than the storage attached to brokers allows. You will get acquainted with the different configuration options and know what to expect when you enable the feature, like for example when will the first upload to the remote object storage take place.
Building a Self-Service Stream Processing Portal: How And Why
"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work.
However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business.
You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want.
Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware.
What you’ll take away from our talk:
- What were the pain points in the previous environment
- How we transitioned to Confluent without service downtime
- Creating a self-service stream processing portal built on top of Connect and ksqlDB
- Use case of stream process portal"
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics.
Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application.
At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."
Navigating Private Network Connectivity Options for Kafka Clusters
"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option!
In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met.
I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines.
Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"
Explaining How Real-Time GenAI Works in a Noisy Pub
"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many.
LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data.
Large language models for your streaming application - explained with a little maths and a lot of pub stories"
TL;DR Kafka Metrics | Kafka Summit London
"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them.
This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."
A Window Into Your Kafka Streams Tasks | KSL
Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance.
- Introduction to Kafka producer internal and workflow
- Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance
- Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient.
- Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees.
- Q&A session with attendees to address specific questions and concerns."
Data Contracts Management: Schema Registry and Beyond
"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element.
In this talk, we’ll:
1. see why data contracts are so important but also difficult to implement;
2. identify the characteristics of a well-designed data contract:
discuss the anatomy of a data contract, its main elements and, how to formally describe them;
3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."
Code-First Approach: Crafting Efficient Flink Apps
"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications.
We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications.
Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."
Debezium vs. the World: An Overview of the CDC Ecosystem
"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you!
[1] https://debezium.io/"
Beyond Tiered Storage: Serverless Kafka with No Local Disks
"Separation of compute and storage has become the de-facto standard in the data industry for batch processing.
The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world.
In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks.
Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it.
This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."
More from HostedbyConfluent (20)
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Evolution of NRT Data Ingestion Pipeline at Trendyol
Evolution of NRT Data Ingestion Pipeline at Trendyol
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques
Exactly-once Stream Processing with Arroyo and Kafka
Exactly-once Stream Processing with Arroyo and Kafka
Building a Self-Service Stream Processing Portal: How And Why
Building a Self-Service Stream Processing Portal: How And Why
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...
Navigating Private Network Connectivity Options for Kafka Clusters
Navigating Private Network Connectivity Options for Kafka Clusters
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Apache Flink: Building a Company-wide Self-service Streaming Data Platform
Explaining How Real-Time GenAI Works in a Noisy Pub
Explaining How Real-Time GenAI Works in a Noisy Pub
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Mastering Kafka Producer Configs: A Guide to Optimizing Performance
Data Contracts Management: Schema Registry and Beyond
Data Contracts Management: Schema Registry and Beyond
Code-First Approach: Crafting Efficient Flink Apps
Code-First Approach: Crafting Efficient Flink Apps
Debezium vs. the World: An Overview of the CDC Ecosystem
Debezium vs. the World: An Overview of the CDC Ecosystem
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Beyond Tiered Storage: Serverless Kafka with No Local Disks
Recently uploaded
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
When it comes to unit testing in the .NET ecosystem, developers have a wide range of options available. Among the most popular choices are NUnit, XUnit, and MSTest. These unit testing frameworks provide essential tools and features to help ensure the quality and reliability of code. However, understanding the differences between these frameworks is crucial for selecting the most suitable one for your projects.
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Finale of the Year: Apply for Next One!
Presentation for the event called "Finale of the Year: Apply for Next One!" organized by GDSC PJATK
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
AWS Cloud Cost Optimization Presentation.pptx
This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.
Azure API Management to expose backend services securely
How to use Azure API Management to expose backend service securely
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Digital Marketing Trends in 2024 | Guide for Staying Ahead
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Generating privacy-protected synthetic data using Secludy and Milvus
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Trusted Execution Environment for Decentralized Process Mining
Presentation of the paper "Trusted Execution Environment for Decentralized Process Mining" given during the CAiSE 2024 Conference in Cyprus on June 7, 2024.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Fueling AI with Great Data with Airbyte Webinar
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Recently uploaded (20)
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdf
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Azure API Management to expose backend services securely
Azure API Management to expose backend services securely
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Digital Marketing Trends in 2024 | Guide for Staying Ahead
Generating privacy-protected synthetic data using Secludy and Milvus
Generating privacy-protected synthetic data using Secludy and Milvus
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Trusted Execution Environment for Decentralized Process Mining
Trusted Execution Environment for Decentralized Process Mining
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Restate: Stream Processing, but for Microservices
- 1. 1 Stream Processing… … but for microservices Giselle van Dongen Stephan Ewen https://restate.dev/
- 2. 2
- 3. 3 The realm of stream processing: Flink, Kafka, Materialize, and friends
- 5. 5 What makes this hard? Retries/run-to-completion State consistency Dual-/multi-write problems Observability Complex infra (MQ, KV, Service Mesh, Workflows Manager, …) Duplicates / idempotency Delayed requests/events Reliable timeouts Manual state machines, request races
- 6. The dangers of not getting this right? 6
- 7. Every sufficiently complex event-driven app eventually becomes a (bad) stream processor implementation. Every sufficiently complex request-based service eventually becomes a (bad) workflow engine. 7 Greenspan’s (lesser-known) 11th rule:
- 8. 8 Resilient applications using distributed durable async/await. Think functions and futures/promises, but distributed and fault-tolerant. Powered by a fast, yet lightweight, event-driven foundation
- 9. 9 service A service B service C gRPC HTTP Kafka SDK SDK SDK timers durable execution workflows reliable RPC K/V store Restate: An event-broker on steroids
- 10. Comparison: Event-processing with Kafka 10
- 11. Comparison: Event-processing with Restate 11
- 18. Order status Restaurant PoS System Payment Provider driver customer Delivery Driver pool order API GW driver-updates Driver digital twin Order workflow https://demo.restate.dev/ Demo: order processing
- 19. Order status Restaurant PoS System Payment Provider driver customer Delivery Driver pool API GW driver-updates Driver digital twin https://demo.restate.dev/ Demo: order processing RPC durable side effects persistent promises durable timers Order workflow Kafka events triggering serverless workflows order
- 20. Example Use Cases for Restate 20 Long-running & low-latency workflows as code Stateful Serverless apps/services Microservice orchestration Background tasks, task queues, cron jobs Control planes
- 21. Revisiting… “Stream Processing is a Protocol” 21 https://twitter.com/jaykreps/status/925761011952467968
- 22. … but: the protocol should be different 22 How many implementations of KStreams? Complexity of large state (RocksDB), timers, back-pressure, etc. PoC stream processing lib for Restate in a few 100s of lines of code. Limited: Few transformations, low throughput, no back-pressure, no event time. But: Exactly-once, low latency, large state, zero-downtime upgrades, fast scaling of compute, dynamic parallelism, state queries, tracing. How about this protocol:
- 23. 23 Want to try out Restate? Sign up for beta access: https://restate.dev/ @restatedev /company/restatedev Easily build resilient applications using distributed durable async/await. ● Durable execution, resilient, suspendable ● Stateful and Serverless ● Reliable messaging / RPC ● Persistent Promises to connect systems ● Persistent timers ● SQL over distributed status and state ● Tracing / OTEL ● Bridge Kafka → FaaS ● Bridge Events and RPC world ● Stateful serverless workflows as code ● Resilient microservices / orchestration ● Event-driven applications ● Control Planes