Wix has finally released to open-source its Kafka client SDK wrapper called Greyhound.
Completely re-written using the Scala functional library ZIO.
Greyhound harnesses ZIO's sophisticated async and concurrency features together with its easy composability to provide a superior experience to Kafka's own client SDKs.
It offers rich functionality including:
- Trivial setup of message processing parallelisation,
- Various fault tolerant retry policies (for consumers AND producers),
- Easy plug-ability of metrics publishing and context propagation and much more.
This talk will also show how Greyhound is used by Wix developers in more than 1000 event-driven microservices.
Exactly Once Delivery with Kafka - Kafka Tel-Aviv MeetupNatan Silnitsky
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
Wix has finally released to open-source its Kafka client SDK wrapper called Greyhound.
Completely re-written using the Scala functional library ZIO.
Greyhound harnesses ZIO's sophisticated async and concurrency features together with its easy composability to provide a superior experience to Kafka's own client SDKs.
It offers rich functionality including:
- Trivial setup of message processing parallelisation,
- Various fault tolerant retry policies (for consumers AND producers),
- Easy plug-ability of metrics publishing and context propagation and much more.
This talk will also show how Greyhound is used by Wix developers in more than 2000 event-driven microservices.
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
Exactly Once Delivery with Kafka - JOTB2020 Mini SessionNatan Silnitsky
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
10 Lessons Learned from using Kafka with 1000 microservices - java global summitNatan Silnitsky
Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices.
We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place.
In these slides you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include:
* How to increase dev velocity of event driven style code.
* How to optimize working with Kafka in polyglot setting
* How to support growing amount of traffic and developers.
* How to tackle multiple DCs environment.
Advanced Caching Patterns used by 2000 microservices - Api WorldNatan Silnitsky
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various tools and features for adapting to growing scale and insuring high resilience.
Jax london - Battle-tested event-driven patterns for your microservices archi...Natan Silnitsky
During the past couple of years I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs on top of Kafka that have facilitated creating a robust distributed microservices system at Wix that can easily handle increasing traffic and storage needs with many different use-cases.
In this talk I will share these patterns with you, including:
* Consume and Project (data decoupling)
* End-to-end Events (Kafka+websockets)
* In memory KV stores (consume and query with 0-latency)
* Events transactions (Exactly Once Delivery)
Exactly Once Delivery with Kafka - Kafka Tel-Aviv MeetupNatan Silnitsky
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
Wix has finally released to open-source its Kafka client SDK wrapper called Greyhound.
Completely re-written using the Scala functional library ZIO.
Greyhound harnesses ZIO's sophisticated async and concurrency features together with its easy composability to provide a superior experience to Kafka's own client SDKs.
It offers rich functionality including:
- Trivial setup of message processing parallelisation,
- Various fault tolerant retry policies (for consumers AND producers),
- Easy plug-ability of metrics publishing and context propagation and much more.
This talk will also show how Greyhound is used by Wix developers in more than 2000 event-driven microservices.
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
Exactly Once Delivery with Kafka - JOTB2020 Mini SessionNatan Silnitsky
In this talk I go over the basic theory of messaging in distributed systems, the different message delivery guarantees in Kafka and the to use them.
I focus on exactly once delivery guarantees and the way Kafka implements it with transaction based messaging protocol.
Including a discussion of the latency/throughput trade-offs, resource utilisation and its overall advantages and shortcomings.
Finally, I show a use-case at Wix where exactly once delivery helped us solve a big problem.
10 Lessons Learned from using Kafka with 1000 microservices - java global summitNatan Silnitsky
Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 microservices.
We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place.
In these slides you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include:
* How to increase dev velocity of event driven style code.
* How to optimize working with Kafka in polyglot setting
* How to support growing amount of traffic and developers.
* How to tackle multiple DCs environment.
Advanced Caching Patterns used by 2000 microservices - Api WorldNatan Silnitsky
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various tools and features for adapting to growing scale and insuring high resilience.
Jax london - Battle-tested event-driven patterns for your microservices archi...Natan Silnitsky
During the past couple of years I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs on top of Kafka that have facilitated creating a robust distributed microservices system at Wix that can easily handle increasing traffic and storage needs with many different use-cases.
In this talk I will share these patterns with you, including:
* Consume and Project (data decoupling)
* End-to-end Events (Kafka+websockets)
* In memory KV stores (consume and query with 0-latency)
* Events transactions (Exactly Once Delivery)
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUANatan Silnitsky
Kafka is the bedrock of Wix’s distributed Mega Microservices system.
Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices.
In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system.
These Include:
* How to increase dev velocity of event-driven style code.
* How to optimize working with Kafka in polyglot setting
* How to migrate from request-reply to event-driven
* How to tackle multiple DCs environment.
How to build 1000 microservices with Kafka and thriveNatan Silnitsky
This talk is about the Wix ecosystem for event driven architecture on top of Kafka.
I share the best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...Natan Silnitsky
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet, StatefulSet) or even deploy a sidecar.
Battle Tested Event-Driven Patterns for your Microservices Architecture - Ris...Natan Silnitsky
During the past couple of years I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs on top of Kafka that have facilitated creating a robust distributed microservices system at Wix that can easily handle increasing traffic and storage needs with many different use-cases.
In this talk I will share these patterns with you, including:
* Consume and Project (data decoupling)
* End-to-end Events (Kafka+websockets)
* In memory KV stores (consume and query with 0-latency)
* Events transactions (Exactly Once Delivery)
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.
Making Sense of Your Event-Driven Dataflows (Jorge Esteban Quilcate Otoya, SY...confluent
Contrary to RPC-like applications, where communication and dependencies are explicitly defined between Services; data flowing between Event-Driven Applications is defined by how do they react to and emit events. A trade-off between Data-flow explicitness and Service autonomy becomes apparent between this two architectural-styles. The goal in this presentation is to demonstrate how Distributed-Tracing can help to cope with this trade-off, turning messaging exchange between decoupled, autonomous, Event-Driven Services, into explicit Data-flows. Zipkin project brings a Distributed-Tracing infrastructure that enables the collection, processing, and visualization of traces produced by RPC-based, as well as messaging-based applications.
This presentation includes demonstrations on how to enable Tracing for Kafka Streams applications, Kafka Connectors, and KSQL; evidencing how implicit Services behavior and communication through the event-log become can become explicit via Distributed-Tracing. But collecting and visualizing traces is just the first step. In order to create insights from tracing-data, models has to be built to enable an better understanding from the system, and improve our operational capabilities. Including research-based experiences from Netflix[1] and Facebook[2] on how tracing-data has been processed and polished with multiple purposes, this presentation will cover how service-dependency analysis and anomaly-detection models can be built on top of it.
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative Natan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
WebSocket MicroService vs. REST MicroserviceRick Hightower
Comparing the speed of RPC calls over WebScoket Microservices versus REST based microservices. Using wrk, QBit, and examples in Java we show how much faster WebSocket is for doing RPC service calls.
Open sourcing a successful internal project - Reversim 2021Natan Silnitsky
About a year ago data streams team at Wix has released to open-source its Kafka client SDK wrapper called Greyhound.
Greyhound offers rich functionality like message processing parallelisation and batching, various fault tolerant retry policies and much more.
This talk will show how the team designed Greyhound with a layered architecture to allow both public and private parts and also different levels of flexible configuration.
How it automatically syncs only relevant code from private repo to public one and also how it securely accepts public PRs back to the private repo.
Outline:
* Quick intro on what Greyhound is and its history at Wix
* Greyhound layered architecture design to allow both public and private parts and also different levels of flexible configuration.
* How it automatically syncs only relevant code from private repo to public one using Copybara tool
* how it securely accepts public PRs back to the private repo.
Using Microservices Architecture and Patterns to Address Applications Require...Prem Sankar Gopannan
Gives an overview about MEC application requirements and microservice patters. Provides a sample app and application of patterns to better design the MEC app. Provides an overview about state management of MEC app
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020HostedbyConfluent
What does a Kafka administrator need to do if they have a user who demands that message delivery be guaranteed, fast, and low cost? In this talk we walk through the architecture we created to deliver for such users. Learn around the alternatives we considered and the pros and cons around what we came up with.
In this talk, we’ll be forced to dive into broker restart and failure scenarios and things we need to do to prevent leader elections from slowing down incoming requests. We’ll need to take care of the consumers as well to ensure that they don’t process the same request twice. We also plan to describe our architecture by showing a demo of simulated requests being produced into Kafka clusters and consumers processing them in lieu of us aggressively causing failures on the Kafka clusters.
We hope the audience walks away with a deeper understanding of what it takes to build robust Kafka clients and how to tune them to accomplish stringent delivery guarantees.
Multi-Clusters Made Easy with Liqo: Getting Rid of Your Clusters Keeping Them...KCDItaly
Many companies are experiencing a dramatic increase in the number of their Kubernetes clusters, for
reasons such as geographical/legislative constraints, data/service replication, etc.
However, when the number of clusters increases, the complexity of deploying apps, managing the entire
multi-cluster infrastructure, and keeping its state under control, becomes rapidly an unmanageable
problem.
A possible solution is Liqo, an open-source project that simplifies the creation of multi-cluster topologies
by replicating the Kubernetes “cattle” model also to clusters.
Liqo creates a virtual cluster that spans multiple real clusters, either on-prem or managed (AKS, EKS,
GKE), and instantiates the desired applications seamlessly in the appropriate cluster.
This talk will discuss the potentials and roadblocks of this vision and highlight how Liqo brings multi-
cluster transparency to the users.
Mariusz Richtscheid: Are you wondering about using serverless computing in your project? In this presentation, I will explain the basics of serverless architecture. I'm using an example of a real-life serverless project done at The Software House called Babelsheet (translation-management solution), and include many practical tips for any beginner starting their adventure with a serverless architecture.
Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov ...confluent
When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. NOTE: This talk will be delivered with Michael Ng, product manager, Confluent
Introducing envoy-based service mesh at Booking.comIvan Kruglov
Service mesh is a dedicated layer of a company's infrastructure which should simplify communication between services and make it reliable, secure and observable.
In this talk, we'll go deep into Booking.com's case study of introducing service mesh. We will discover the reasons and objectives of the project. Why Envoy was selected as the base rather than other available options. Find out what is the setup and features of the homegrown control plane. We will expand on what service is provided for developers and how they safely deploy potentially dangerous configuration changes. Finally, we will talk about pitfalls met along the way.
Over the past few years, web-applications have started to play an increasingly important role in our lives. We expect them to be always available and the data to be always fresh. This shift into the realm of real-time data processing is now transitioning to physical devices, and Gartner predicts that the Internet of Things will grow to an installed base of 26 billion units by 2020.
Reactive web-applications are an answer to the new requirements of high-availability and resource efficiency brought by this rapid evolution. On the JVM, a set of new languages and tools has emerged that enable the development of entirely asynchronous request and data handling pipelines. At the same time, container-less application frameworks are gaining increasing popularity over traditional deployment mechanisms.
This talk is going to give you an introduction into one of the most trending reactive web-application stack on the JVM, involving the Scala programming language, the concurrency toolkit Akka and the web-application framework Play. It will show you how functional programming techniques enable asynchronous programming, and how those technologies help to build robust and resilient web-applications.
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUANatan Silnitsky
Kafka is the bedrock of Wix’s distributed Mega Microservices system.
Over the years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1400 mostly Scala microservices.
In this talk, you will learn about 10 key decisions and steps you can take in order to safely scale-up your Kafka-based system.
These Include:
* How to increase dev velocity of event-driven style code.
* How to optimize working with Kafka in polyglot setting
* How to migrate from request-reply to event-driven
* How to tackle multiple DCs environment.
How to build 1000 microservices with Kafka and thriveNatan Silnitsky
This talk is about the Wix ecosystem for event driven architecture on top of Kafka.
I share the best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.
Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...Natan Silnitsky
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet, StatefulSet) or even deploy a sidecar.
Battle Tested Event-Driven Patterns for your Microservices Architecture - Ris...Natan Silnitsky
During the past couple of years I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs on top of Kafka that have facilitated creating a robust distributed microservices system at Wix that can easily handle increasing traffic and storage needs with many different use-cases.
In this talk I will share these patterns with you, including:
* Consume and Project (data decoupling)
* End-to-end Events (Kafka+websockets)
* In memory KV stores (consume and query with 0-latency)
* Events transactions (Exactly Once Delivery)
Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky
At Wix, we have created a universal event-driven programming infrastructure on top of the Kafka message broker.
This infra makes sure messages are eventually successfully consumed and produced no matter what failure it encounters.
In this talk, you will learn about the features we introduced in order to make sure our distributed system can safely handle an ever growing message throughput in a fault tolerant manner.
You will be introduced to such techniques as retry topics, local persistent queues, and cooperative fibers that help make your flows more resilient and performant.
You will also learn how to make this infra work for all programming languages tech stacks with optimal resource manage using the power of Kubernetes and gRPC.
When to use a client library, and when to deploy an external pod (DaemonSet) or even deploy a sidecar.
Making Sense of Your Event-Driven Dataflows (Jorge Esteban Quilcate Otoya, SY...confluent
Contrary to RPC-like applications, where communication and dependencies are explicitly defined between Services; data flowing between Event-Driven Applications is defined by how do they react to and emit events. A trade-off between Data-flow explicitness and Service autonomy becomes apparent between this two architectural-styles. The goal in this presentation is to demonstrate how Distributed-Tracing can help to cope with this trade-off, turning messaging exchange between decoupled, autonomous, Event-Driven Services, into explicit Data-flows. Zipkin project brings a Distributed-Tracing infrastructure that enables the collection, processing, and visualization of traces produced by RPC-based, as well as messaging-based applications.
This presentation includes demonstrations on how to enable Tracing for Kafka Streams applications, Kafka Connectors, and KSQL; evidencing how implicit Services behavior and communication through the event-log become can become explicit via Distributed-Tracing. But collecting and visualizing traces is just the first step. In order to create insights from tracing-data, models has to be built to enable an better understanding from the system, and improve our operational capabilities. Including research-based experiences from Netflix[1] and Facebook[2] on how tracing-data has been processed and polished with multiple purposes, this presentation will cover how service-dependency analysis and anomaly-detection models can be built on top of it.
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative Natan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
WebSocket MicroService vs. REST MicroserviceRick Hightower
Comparing the speed of RPC calls over WebScoket Microservices versus REST based microservices. Using wrk, QBit, and examples in Java we show how much faster WebSocket is for doing RPC service calls.
Open sourcing a successful internal project - Reversim 2021Natan Silnitsky
About a year ago data streams team at Wix has released to open-source its Kafka client SDK wrapper called Greyhound.
Greyhound offers rich functionality like message processing parallelisation and batching, various fault tolerant retry policies and much more.
This talk will show how the team designed Greyhound with a layered architecture to allow both public and private parts and also different levels of flexible configuration.
How it automatically syncs only relevant code from private repo to public one and also how it securely accepts public PRs back to the private repo.
Outline:
* Quick intro on what Greyhound is and its history at Wix
* Greyhound layered architecture design to allow both public and private parts and also different levels of flexible configuration.
* How it automatically syncs only relevant code from private repo to public one using Copybara tool
* how it securely accepts public PRs back to the private repo.
Using Microservices Architecture and Patterns to Address Applications Require...Prem Sankar Gopannan
Gives an overview about MEC application requirements and microservice patters. Provides a sample app and application of patterns to better design the MEC app. Provides an overview about state management of MEC app
Can Kafka Handle a Lyft Ride? (Andrey Falko & Can Cecen, Lyft) Kafka Summit 2020HostedbyConfluent
What does a Kafka administrator need to do if they have a user who demands that message delivery be guaranteed, fast, and low cost? In this talk we walk through the architecture we created to deliver for such users. Learn around the alternatives we considered and the pros and cons around what we came up with.
In this talk, we’ll be forced to dive into broker restart and failure scenarios and things we need to do to prevent leader elections from slowing down incoming requests. We’ll need to take care of the consumers as well to ensure that they don’t process the same request twice. We also plan to describe our architecture by showing a demo of simulated requests being produced into Kafka clusters and consumers processing them in lieu of us aggressively causing failures on the Kafka clusters.
We hope the audience walks away with a deeper understanding of what it takes to build robust Kafka clients and how to tune them to accomplish stringent delivery guarantees.
Multi-Clusters Made Easy with Liqo: Getting Rid of Your Clusters Keeping Them...KCDItaly
Many companies are experiencing a dramatic increase in the number of their Kubernetes clusters, for
reasons such as geographical/legislative constraints, data/service replication, etc.
However, when the number of clusters increases, the complexity of deploying apps, managing the entire
multi-cluster infrastructure, and keeping its state under control, becomes rapidly an unmanageable
problem.
A possible solution is Liqo, an open-source project that simplifies the creation of multi-cluster topologies
by replicating the Kubernetes “cattle” model also to clusters.
Liqo creates a virtual cluster that spans multiple real clusters, either on-prem or managed (AKS, EKS,
GKE), and instantiates the desired applications seamlessly in the appropriate cluster.
This talk will discuss the potentials and roadblocks of this vision and highlight how Liqo brings multi-
cluster transparency to the users.
Mariusz Richtscheid: Are you wondering about using serverless computing in your project? In this presentation, I will explain the basics of serverless architecture. I'm using an example of a real-life serverless project done at The Software House called Babelsheet (translation-management solution), and include many practical tips for any beginner starting their adventure with a serverless architecture.
Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov ...confluent
When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. NOTE: This talk will be delivered with Michael Ng, product manager, Confluent
Introducing envoy-based service mesh at Booking.comIvan Kruglov
Service mesh is a dedicated layer of a company's infrastructure which should simplify communication between services and make it reliable, secure and observable.
In this talk, we'll go deep into Booking.com's case study of introducing service mesh. We will discover the reasons and objectives of the project. Why Envoy was selected as the base rather than other available options. Find out what is the setup and features of the homegrown control plane. We will expand on what service is provided for developers and how they safely deploy potentially dangerous configuration changes. Finally, we will talk about pitfalls met along the way.
Over the past few years, web-applications have started to play an increasingly important role in our lives. We expect them to be always available and the data to be always fresh. This shift into the realm of real-time data processing is now transitioning to physical devices, and Gartner predicts that the Internet of Things will grow to an installed base of 26 billion units by 2020.
Reactive web-applications are an answer to the new requirements of high-availability and resource efficiency brought by this rapid evolution. On the JVM, a set of new languages and tools has emerged that enable the development of entirely asynchronous request and data handling pipelines. At the same time, container-less application frameworks are gaining increasing popularity over traditional deployment mechanisms.
This talk is going to give you an introduction into one of the most trending reactive web-application stack on the JVM, involving the Scala programming language, the concurrency toolkit Akka and the web-application framework Play. It will show you how functional programming techniques enable asynchronous programming, and how those technologies help to build robust and resilient web-applications.
UK Symfony Meetup, November 2012
Original presentation: https://docs.google.com/presentation/pub?id=1PLcqTby6yqSbfWlMIDHknH852DU6DO6OAgQJOtSEdsg&start=false&loop=false&delayms=3000
8 Lessons Learned from Using Kafka in 1500 microservices - confluent streamin...Natan Silnitsky
Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1500 microservices.
We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place.
In these slides you will learn about 8 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include:
* How to increase dev velocity of event driven style code.
* How to optimize working with Kafka in polyglot setting
* How to support growing amount of traffic and developers.
Adding 1.21 Gigawatts to Applications with RabbitMQ (PHPNW Dec 2014 Meetup)James Titcumb
As your application grows, you soon realise you need to break up your application into smaller chunks that talk to each other. You could just use web services to interact, or you could take a more robust approach and use the message broker RabbitMQ. In this talk, we will take a look at the techniques you can use to vastly enhance inter-application communication, learn about the core concepts of RabbitMQ, cover how you can scale different parts of your application separately, and modernise your development using a message-oriented architecture
Deep dive in container service discoveryDocker, Inc.
Service discovery and traffic load-balancing in the container ecosystem relies on different technologies, such as IPVS and iptables, and container orchestrators use different approaches. This talk will present in details how Docker Swarm and Kubernetes achieve this. The talk will continue with a demo showing how applications that are not managed by Kubernetes can take advantage of its native load-balancing. Finally, it will compare these approaches to service-mesh solutions.
Open Source Summit 2018, Vancouver (Canada): Workshop by Josef Adersberger (@adersberger, CTO at QAware) and Michael Frank (Software Architect at QAware)
Abstract:
Istio service mesh is a thrilling new tech that helps getting a lot of technical stuff out of your microservices (circuit breaking, observability, mutual-TLS, ...) into the infrastructure - for those who are lazy (aka productive) and want to keep their microservices small. Come one, come all to the Istio playground:
(1) We provide an overview of all current Istio features on a YAML and CLI level.
(2) We guide you through the installation of Istio on a local Kubernetes cluster.
(3) We bring a small sample application.
(4) We provide assistance in the case you get stuck ... and it's up to you to explore and tinker with Istio on your own paths and with your own pace.
*** Please find prerequisites and content here: https://github.com/adersberger/istio-playground ***
Getting up to speed with Kafka Connect: from the basics to the latest feature...HostedbyConfluent
"Kafka Connect is an ideal tool for building data pipelines. It is both reliable and scalable, with a pluggable interface that lets you flow data between Kafka and any system you need. A Connect pipeline is made up of many different components, and understanding how each of these interact together is essential, even for the simplest setup.
In this talk we will introduce the Connect components, from connectors, to transformations to the runtime itself. We will also share some of the new capabilities and best practices that you should be aware of to help you run and manage connectors effectively.
Finally we will talk about some different open source projects that have been built on top of Connect that can help you get the most out of the framework."
Kafka as a Platform: the Ecosystem from the Ground Up with Robin Moffatt | Ka...HostedbyConfluent
Apache Kafka® has become a key data infrastructure technology, and we all have at least a vague sense that it is a messaging system, but what else is it? How can an overgrown message bus be getting this much buzz? Well, because Kafka is merely the center of a rich streaming data platform that invites detailed exploration.
In this talk, we’ll look at the entire streaming platform provided by Apache Kafka and the Confluent community components. Starting with a lonely key-value pair, we’ll build up topics, partitioning, replication, and low-level Producer and Consumer APIs. We’ll group consumers into elastically scalable, fault-tolerant application clusters, then layer on more sophisticated stream processing APIs like Kafka Streams and ksqlDB. We’ll help teams collaborate around data formats with schema management. We’ll integrate with legacy systems without writing custom code. By the time we’re done, the open-source project we thought was Big Data’s answer to message queues will have become an enterprise-grade streaming platform, all in 45 minutes!
A year ago, I was a committed VPS and dedicated-machine deployer. I thought the cloud imposed silly restrictions - how dare you take away my shell account! Whaddya mean I can't save files locally? Since then, I've had some interesting experiences. I've worked on big cloud-deployed systems, and certain large traditionally-deployed systems, and I've seen how a lot of the decisions that you're ... encouraged to make when designing an app to run in the cloud. Most interestingly, I've discovered how those same decisions can make for a much better app regardless of where it'll end up. In this talk, I'll share those architectural patterns with you, and show why they work. Hopefully, I'll convince all of you to build cloud castles -- even if you've got your foundation firmly on the ground.
In this meetup, Liran Cohen, Cloud platform & DevOps Team Leader, will talk about some of Kubernetes key concepts. We will learn about the architecture of the system; the different resources available in the system; the problems it’s trying to solve, and the model that it uses to manage containerized application deployments.
8 Lessons Learned from Using Kafka in 1000 Scala microservices - Scale by the...Natan Silnitsky
Kafka is the bedrock of Wix's distributed microservices system. For the last 5 years we have learned a lot about how to successfully scale our event-driven architecture to roughly 1500 microservices.
We’ve managed to achieve higher decoupling and independence for our various services and dev teams that have very different use-cases while maintaining a single uniform infrastructure in place.
In these slides you will learn about 8 key decisions and steps you can take in order to safely scale-up your Kafka-based system. These include:
* How to increase dev velocity of event driven style code.
* How to optimize working with Kafka in polyglot setting
* How to support growing amount of traffic and developers.
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Effective Strategies for Wix's Scaling challenges - GeeConNatan Silnitsky
This session unveils the multifaceted horizontal scaling strategies that power Wix's robust infrastructure. From Kafka consumer scaling and dynamic traffic routing for site segments to DynamoDB sharding and MySQL clusters custom routing with ProxySQL, we dissect the mechanisms that ensure scalability and performance at Wix.
Attendees will learn about the art of sharding and routing key selection across different systems, and how to apply these strategies to their own infrastructure. We'll share insights into choosing the right scaling strategy for various scenarios, balancing between managed services and custom solutions.
Key Takeaways:
- Grasp various sharding techniques and routing strategies used at Wix.
- Understand key considerations for sharding key and routing rule selection.
- Learn when and why to choose specific horizontal scaling strategies.
- Gain practical knowledge for applying these strategies to achieve scalability and high availability.
Join us to gain a blueprint for scaling your systems horizontally, drawing from Wix's proven practices.
Discover how Wix transitioned from complex event sourcing and CQRS to streamlined CRUD services, optimizing their vast platform for better scalability, performance, and resiliency.
Wix's platform, designed to accommodate diverse business needs, boasts:
* 3.5 Billion daily HTTP transactions
* 70 Billion Kafka messages per day
* Roughly 4000 microservices in production
This session will highlight the simplification of Wix's architecture through domain events, resilient Kafka messaging, and advanced techniques like materialization and caching. By standardizing APIs and employing tools like protobuf and gRPC, Wix has enhanced the developer experience, both internally and externally, and fostered an open, integrative platform.
Attendees will gain insights into Wix's strategies for microservice coordination, ensuring system resilience and data consistency, as well as query performance optimization through innovative 2-level caching solutions.
Workflow Engines & Event Streaming Brokers - Can they work together? [Current...Natan Silnitsky
Workflow engines and event streaming brokers offer very different solutions to the same requirement - an optimal implementation of microservices communication.
At Wix, we have a good experience with event-driven architecture for our 2500 microservices using Apache Kafka. Apache Kafka provides:
* support for very high throughput
* Fault tolerance
* very loose coupling
* Huge connectors eco-system
Temporal workflow orchestration has interesting features:
* Support for long running tasks
* business flows visual tracking
* Easy to follow imperative style programming
In this talk we will learn about the tradeoffs between the two technologies and how to implement various use cases in each architecture, including those that need a little more work.
DevSum - Lessons Learned from 2000 microservicesNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
GeeCon - Lessons Learned from 2000 microservicesNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Migrating to Multi Cluster Managed Kafka - ApacheKafkaILNatan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
BuildStuff - Lessons Learned from 2000 Event Driven MicroservicesNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Lessons Learned from 2000 Event Driven Microservices - ReversimNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Devoxx Ukraine - Kafka based Global Data MeshNatan Silnitsky
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Devoxx UK - Migrating to Multi Cluster Managed KafkaNatan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Dev Days Europe - Kafka based Global Data Mesh at WixNatan Silnitsky
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Kafka Summit London - Kafka based Global Data Mesh at WixNatan Silnitsky
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
5 Takeaways from Migrating a Library to Scala 3 - Scala LoveNatan Silnitsky
Scala 3 is going to make Scala more easy to write, and especially read. more power features like enums and less misleading keywords like implicit.
But first we need to migrate our old Scala 2.12 / 2.13 codebase to Scala 3.
This talk tells the story of how I tried to migrate greyhound open-source library to Scala 3 with partial success.
You will hear about what works, what doesn't and about a few pitfalls to avoid
Migration takeaways include:
1. use migration tools, don't do it manually
2. which popular 3rd party libraries can and can't be used by Scala 3 code
and many more
Migrating to Multi Cluster Managed Kafka - DevopStars 2022Natan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
How to successfully manage a ZIO fiber’s lifecycle - Functional Scala 2021Natan Silnitsky
Fibers are the backbone of the highly performant, asynchronous and concurrent abilities of ZIO. They are lightweight “green threads” implemented by the ZIO runtime system.
In this lightening talk you will learn about:
* How to handle fiber dying due to unexpected failure
* How to guarantee a ZIO fiber is interrupted
* How to set up fiber tracking and executing a fiberDump for increased debuggability
Advanced Caching Patterns used by 2000 microservices - Code MotionNatan Silnitsky
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
Advanced Caching Patterns used by 2000 microservices - Devoxx UkraineNatan Silnitsky
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
Battle-tested event-driven patterns for your microservices architecture - Sca...Natan Silnitsky
During the past couple of years I’ve implemented or have witnessed implementations of several key patterns of event-driven messaging designs on top of Kafka that have facilitated creating a robust distributed microservices system at Wix that can easily handle increasing traffic and storage needs with many different use-cases.
In this talk I will share these patterns with you, including:
* Consume and Project (data decoupling)
* End-to-end Events (Kafka+websockets)
* In memory KV stores (consume and query with 0-latency)
* Events transactions (Exactly Once Delivery)
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Welocme to ViralQR, your best QR code generator.ViralQR
Welcome to ViralQR, your best QR code generator available on the market!
At ViralQR, we design static and dynamic QR codes. Our mission is to make business operations easier and customer engagement more powerful through the use of QR technology. Be it a small-scale business or a huge enterprise, our easy-to-use platform provides multiple choices that can be tailored according to your company's branding and marketing strategies.
Our Vision
We are here to make the process of creating QR codes easy and smooth, thus enhancing customer interaction and making business more fluid. We very strongly believe in the ability of QR codes to change the world for businesses in their interaction with customers and are set on making that technology accessible and usable far and wide.
Our Achievements
Ever since its inception, we have successfully served many clients by offering QR codes in their marketing, service delivery, and collection of feedback across various industries. Our platform has been recognized for its ease of use and amazing features, which helped a business to make QR codes.
Our Services
At ViralQR, here is a comprehensive suite of services that caters to your very needs:
Static QR Codes: Create free static QR codes. These QR codes are able to store significant information such as URLs, vCards, plain text, emails and SMS, Wi-Fi credentials, and Bitcoin addresses.
Dynamic QR codes: These also have all the advanced features but are subscription-based. They can directly link to PDF files, images, micro-landing pages, social accounts, review forms, business pages, and applications. In addition, they can be branded with CTAs, frames, patterns, colors, and logos to enhance your branding.
Pricing and Packages
Additionally, there is a 14-day free offer to ViralQR, which is an exceptional opportunity for new users to take a feel of this platform. One can easily subscribe from there and experience the full dynamic of using QR codes. The subscription plans are not only meant for business; they are priced very flexibly so that literally every business could afford to benefit from our service.
Why choose us?
ViralQR will provide services for marketing, advertising, catering, retail, and the like. The QR codes can be posted on fliers, packaging, merchandise, and banners, as well as to substitute for cash and cards in a restaurant or coffee shop. With QR codes integrated into your business, improve customer engagement and streamline operations.
Comprehensive Analytics
Subscribers of ViralQR receive detailed analytics and tracking tools in light of having a view of the core values of QR code performance. Our analytics dashboard shows aggregate views and unique views, as well as detailed information about each impression, including time, device, browser, and estimated location by city and country.
So, thank you for choosing ViralQR; we have an offer of nothing but the best in terms of QR code services to meet business diversity!
9. @NSilnitsky
Kafka Broker
Service A Service B
Abstract
so that it is easy to
change for everyone
Simplify
APIs, with additional
features
Greyhound
wraps Kafka
Kafka
Consumer
Kafka
Producer
10. @NSilnitsky
Multiple APIs
For Java, Scala and
Wix Devs
Greyhound
wraps Kafka
Scala Future ZIO Java
Kafka
Consumer
Kafka
Producer
Kafka Broker
ZIO Core
Service A Service B
* all logic
45. class OldWorker(capacity: Int, /*...*/) {
private val tasksQueue = new LinkedBlockingDeque(capacity)
start()
private def start() = {
// simplified
val thread = new Thread(new TaskLoop)
thread.start()
}
private class TaskLoop extends Runnable {
override def run() = {
// simplified
while (true) {
val task = tasksQueue.take()
task.run()
}
}
}
...
}
Old Worker
46. class OldWorker(capacity: Int, /*...*/) {
private val tasksQueue = new LinkedBlockingDeque(capacity)
start()
private def start() = {
// simplified
val thread = new Thread(new TaskLoop)
thread.start()
}
private class TaskLoop extends Runnable {
override def run() = {
// simplified
while (true) {
val task = tasksQueue.take()
task.run()
}
}
}
...
}
Old Worker
* resource, context, maxPar
47. Greyhound
wraps Kafka
✔ Simple Consumer API
✔ Composable Record
Handler
✔ Parallel Consumption
+ Retries!
What do we
want it to do?
...what about
Error handling?
59. foreachWhile(blockingBackoffs) { interval =>
handleAndBlockWithPolling(interval, blockingStateRef)
}
Current Solution
Checks
blockingState
between short
sleeps
Allows user
request to
unblock
60. foreachWhile(blockingBackoffs) { interval =>
handleAndBlockWithPolling(interval, blockingStateRef)
}
Current Solution
def foreachWhile[R, E, A](as: Iterable[A])(f: A => ZIO[R, E, Boolean]):
ZIO[R, E, Unit] =
ZIO.effectTotal(as.iterator).flatMap { i =>
def loop: ZIO[R, E, Unit] =
if (i.hasNext) f(i.next).flatMap(result => if(result) loop else
ZIO.unit)
else ZIO.unit
loop
}
61. foreachWhile(blockingBackoffs) { interval =>
handleAndBlockWithPolling(interval, blockingStateRef)
}
Current Solution
def foreachWhile[R, E, A](as: Iterable[A])(f: A => ZIO[R, E, Boolean]):
ZIO[R, E, Unit] =
ZIO.effectTotal(as.iterator).flatMap { i =>
def loop: ZIO[R, E, Unit] =
if (i.hasNext) f(i.next).flatMap(result => if(result) loop else
ZIO.unit)
else ZIO.unit
loop
}
Stream.fromIterable(blockingBackoffs).foreachWhile(...)
62. Greyhound
wraps Kafka
✔ Simple Consumer API
✔ Composable Record
Handler
✔ Parallel Consumption
✔ Retries
+ Resilient Producer
What do we
want it to do?
and when
Kafka brokers
are
unavailable...
63. + Retry
on Error
Kafka Broker
Producer
Use Case:
Guarantee completion
Consumer
Wix
Payments
Service
Subscription
renewal
Job
Scheduler
64. + Retry
on Error
Kafka Broker
Producer
Use Case:
Guarantee completion
Consumer
Wix
Payments
Service
Job
Scheduler
68. Greyhound
wraps Kafka
✔ Simple Consumer API
✔ Composable Record
Handler
✔ Parallel Consumption
✔ Retries
✔ Resilient Producer
+ Context Propagation
What do we
want it to do?
Super cool for
us
78. - Much less boilerplate
- Code that’s easier to understand
- Fun
REWRITING
GREYHOUND IN ZIO
RESULTED IN...
79. ...but ZIO offers lower level abstractions too, like Promise and clock.sleep.
SOMETIMES YOU CAN’T DO
EXACTLY WHAT YOU WANT WITH
HIGH LEVEL ZIO OPERATORS