Wix meetup Feb 26 2020 - How to build 1000 microservices with kafka and thrive
•
0 likes•127 views
This talk is about the Wix ecosystem for event driven architecture on top of Kafka. The best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.
Report
Share
Report
Share
Download to read offline
Recommended
How to build 1000 microservices with Kafka and thrive - Wix @Scale Meetup
This talk is about the Wix ecosystem for event driven architecture on top of Kafka.
I share the best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.
Kubernetes + netflix oss
Apresentação realizada no 3º Meetup de cloud native realizado na Ilegra.
Na apresentação é apresentado as funcionalidades oferecidas pelas duas ferramentas e os aprendizados e vantagens destas.
TDC2017 | São Paulo - Trilha Containers How we figured out we had a SRE team ...
Trilha Containers How we figured out we had a SRE team at - Traefik: Dynamic container load balancer for containers
Kubernetes
Manage a cluster of Linux containers as a single system to accelerate Dev and simplify Ops.
Docker Fundamentals
It discusses in brief introduction to Docker, how it differs from Virtual Machine and underlining technologies.
Recommended
How to build 1000 microservices with Kafka and thrive - Wix @Scale Meetup
This talk is about the Wix ecosystem for event driven architecture on top of Kafka.
I share the best practices, SDKs and tools we have created in order to be able to scale our distributed system to more than 1000 microservices.
Kubernetes + netflix oss
Apresentação realizada no 3º Meetup de cloud native realizado na Ilegra.
Na apresentação é apresentado as funcionalidades oferecidas pelas duas ferramentas e os aprendizados e vantagens destas.
TDC2017 | São Paulo - Trilha Containers How we figured out we had a SRE team ...
Trilha Containers How we figured out we had a SRE team at - Traefik: Dynamic container load balancer for containers
Kubernetes
Manage a cluster of Linux containers as a single system to accelerate Dev and simplify Ops.
Docker Fundamentals
It discusses in brief introduction to Docker, how it differs from Virtual Machine and underlining technologies.
Developing Serverless Microservice in Rust
A presentation about my findings while tinkering with possibility of developing serverless microservices using Rust. Presented at Rust Meetup in Mozilla Indonesia, 8 September 2018.
Deploying WSO2 Middleware on Kubernetes
This slide deck will explore the advantages and requirements of Kubernetes. It will also demonstrate how to deploy WSO2 middleware on Kubernetes.
Bucketbench: Benchmarking Container Runtime Performance
A talk presented at the Moby Summit, Los Angeles (a co-located event with the Open Source Summit North America) on Thursday, September 14, 2017. In this talk, an open source tool, bucketbench, was presented as a way to benchmark container runtimes to compare performance impacts of changes in the runtime or changes to the configuration of Docker, runC, or containerd, the three runtimes currently supported in the bucketbench project.
Everything you want to know about Ingress
An Ingress in Kubernetes is a collection of rules that allow inbound connections to reach services. Attend this webinar to learn when to use Ingress controllers. It will cover the types of ingress controllers along with relevant use cases. We will walk
you through a demo of configuring Ingress for a web application.
Serverless architecture: introduction & first steps
Mariusz Richtscheid: Are you wondering about using serverless computing in your project? In this presentation, I will explain the basics of serverless architecture. I'm using an example of a real-life serverless project done at The Software House called Babelsheet (translation-management solution), and include many practical tips for any beginner starting their adventure with a serverless architecture.
LXD Container Hypervisor
LXD is a container "hypervisor" and a new user experience for LXC.
The daemon exports a REST API both locally and if enabled, over the network.
The command line tool is designed to be a very simple, yet very powerful tool to manage all your containers. It can handle connect to multiple container hosts and easily give you an overview of all the containers on your network, let you create some more where you want them and even move them around while they're running.
Kubernetes intro
Kubernetes (commonly referred to as "K8s") is an open-source system for automating deployment, scaling and management of containerized applications It aims to provide a "platform for automating deployment, scaling, and operations of application containers across clusters of hosts". We will see Kubernetes architecture, use cases, basics and live demo
Containerd internals: building a core container runtime
In this talk, we’ll briefly overview of the OpenWhisk serverless (function-as-a-service) framework that initially used the full Docker container engine as the execution vehicle for invoking user functions via containers. After several performance and stability challenges, this project decided to assess the various layers of the Docker engine (containerd and runC) as potential options for the function invoker. Out of that work came an open source project, bucketbench, that can be used to generate benchmarks of container lifecycle operations (e.g., start, stop, kill, remove, pause, unpause) and compare multithreaded operation throughput and stability of each optional engine.
This talk will provide details on the bucketbench project, explain how it has been used to generate performance data for these container runtimes, and shares lessons learned along the way that greatly impact container runtime performance, including bottlenecks in the Linux kernel.
In this talk you’ll learn how you can use bucketbench for your own performance tuning or assessment of container runtimes and how you can collaborate on improvements to the bucketbench project.
Migration of an Enterprise UI Microservice System from Cloud Foundry to Kuber...
Presented at Open Source Summit Japan with Jonathan Schweikhart on June 21, 2018.
Abstract: The 40 Node.js microservices making up the IBM Cloud UI historically have been deployed as apps on Cloud Foundry (CF), an open source PaaS. But, recently, this enterprise microservice system has been migrated to run on Kubernetes to take advantage of improved orchestration, higher availability, and better performance. Tony Erwin & Jonathan Schweikhart will discuss their team's journey and provide you with insights into the advantages of Kube over CF. Even more importantly, they will describe approaches to solving new problems that took the place of old ones, such as: 1) adapting PaaS apps to run as containers on Kube, 2) enabling geo load balancing between the different runtimes (to vette Kube before completely turning off CF), 3) integrating tools like Prometheus into existing monitoring systems, and more! Their team's first-hand experiences will help you avoid pitfalls as you prepare your own migrations to Kube!
Link to Info on Talk: https://ossalsjp18.sched.com/event/EaYj/migration-of-an-enterprise-ui-microservice-system-from-cloud-foundry-to-kubernetes-tony-erwin-jonathan-schweikhart-ibm?iframe=no
NOTE: CF is always evolving and the limitations on private networking and private host names mentioned in the slides are no longer current. If you have access to CF API 2.115.0 or higher (released on June 25, 2018), you can leverage CF's service discovery feature (see https://docs.cloudfoundry.org/devguide/deploy-apps/cf-networking.html#discovery ).
OpenEBS - Containerized Storage for Containers
These slides were presented in the containerization meet-up organized by digital ocean meet-up group in Bangalore. The slides talk about using containers for storage to make the storage truly non-disruptive during upgrades. This is a quick introduction to OpenEBS as well.
Swarm migration
This webinar will walk you through the steps involved in migrating a multi-container application deployed in Docker Swarm to Kubernetes. It will map the concepts of Swarm to Kubernetes. Attend this webinar to learn how to apply your Docker skills to Kubernetes for running and managing containerized applications in production.
Windows server and docker
Types of windows containers
Dockerize the .net application
Using container orchestration engine (Kubernetes) for .Net application
Kubernetes Webinar - Using ConfigMaps & Secrets
Many applications require configuration using some combination of configuration files, command line arguments, and environment variables. ConfigMaps in Kubernetes provide mechanisms to inject containers with configuration data while keeping them portable. Secrets decouple sensitive content from the pods using a volume plug-in. This webinar will discuss the use cases and scenarios for using ConfigMaps and Secrets.
Persistent Storage with Containers with Kubernetes & OpenShift
Manually configuring mounts for containers to various network storage platforms and services is tedious and time consuming. OpenShift and Kubernetes provides a rich library of volume plugins that allow authors of containerized applications (Pods) to declaratively specify what the storage requirements for the containers are so that OpenShift can dynamically provision and allocate the storage assets for the specified containers. As the author of the Kubernetes Persistent Volume specification, I will provide an overview of how Persistent Volume plugins work in OpenShift, demo block storage and file storage volume plugins and close with the Red Hat storage roadmap.
Presented at LinuxCon/ContainerCon by Mark Turansky, Principal Software Engineer, Red Hat
Mark Turansky is a Principal Software Engineer at Red Hat and a full-time contributor to the Kubernetes Project. Mark is the author of the Kubernetes Persistent Volume specification and a member of the Red Hat OpenShift Engineering team.
Kuberenetes - From Zero to Hero
Kubernetes has now become the de facto standard for deploying containerized applications at scale.
The presentation will follow K8s core concepts, architecture and real life scenarios.
Migrating our micro services from Java to Kotlin (Code.Talks 2018)
This talk elaborates our awesome journey migrate all our micro services from Java to Kotlin.
We will take a closer look at the challenges we faced as a team and the impact on our day to day work in an agile environment and how we overcame technical hurdles integrating with Spring Boot and introducing async workflows with Kotlin coroutines.
Finally we discuss at the actual benefits that we gained by moving all our micro services to Kotlin and do a critical review of our journey and an outlook at things to come.
Live Container Migration: OpenStack Summit Barcelona 2016
A talk presented by Phil Estes & Shaun Murakami, IBM Cloud Open Technologies, at the Barcelona OpenStack Summit on October 25, 2016. This talk covers a new feature that will be available in the Docker 1.13 engine for using the CRIU project to checkpoint and restore container processes on Linux. Phil & Shaun present details of this new capability and then demonstrate a proof-of-concept "live migration" of containers across nova compute hosts.
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Effective Strategies for Wix's Scaling challenges - GeeCon
This session unveils the multifaceted horizontal scaling strategies that power Wix's robust infrastructure. From Kafka consumer scaling and dynamic traffic routing for site segments to DynamoDB sharding and MySQL clusters custom routing with ProxySQL, we dissect the mechanisms that ensure scalability and performance at Wix.
Attendees will learn about the art of sharding and routing key selection across different systems, and how to apply these strategies to their own infrastructure. We'll share insights into choosing the right scaling strategy for various scenarios, balancing between managed services and custom solutions.
Key Takeaways:
- Grasp various sharding techniques and routing strategies used at Wix.
- Understand key considerations for sharding key and routing rule selection.
- Learn when and why to choose specific horizontal scaling strategies.
- Gain practical knowledge for applying these strategies to achieve scalability and high availability.
Join us to gain a blueprint for scaling your systems horizontally, drawing from Wix's proven practices.
More Related Content
What's hot
Developing Serverless Microservice in Rust
A presentation about my findings while tinkering with possibility of developing serverless microservices using Rust. Presented at Rust Meetup in Mozilla Indonesia, 8 September 2018.
Deploying WSO2 Middleware on Kubernetes
This slide deck will explore the advantages and requirements of Kubernetes. It will also demonstrate how to deploy WSO2 middleware on Kubernetes.
Bucketbench: Benchmarking Container Runtime Performance
A talk presented at the Moby Summit, Los Angeles (a co-located event with the Open Source Summit North America) on Thursday, September 14, 2017. In this talk, an open source tool, bucketbench, was presented as a way to benchmark container runtimes to compare performance impacts of changes in the runtime or changes to the configuration of Docker, runC, or containerd, the three runtimes currently supported in the bucketbench project.
Everything you want to know about Ingress
An Ingress in Kubernetes is a collection of rules that allow inbound connections to reach services. Attend this webinar to learn when to use Ingress controllers. It will cover the types of ingress controllers along with relevant use cases. We will walk
you through a demo of configuring Ingress for a web application.
Serverless architecture: introduction & first steps
Mariusz Richtscheid: Are you wondering about using serverless computing in your project? In this presentation, I will explain the basics of serverless architecture. I'm using an example of a real-life serverless project done at The Software House called Babelsheet (translation-management solution), and include many practical tips for any beginner starting their adventure with a serverless architecture.
LXD Container Hypervisor
LXD is a container "hypervisor" and a new user experience for LXC.
The daemon exports a REST API both locally and if enabled, over the network.
The command line tool is designed to be a very simple, yet very powerful tool to manage all your containers. It can handle connect to multiple container hosts and easily give you an overview of all the containers on your network, let you create some more where you want them and even move them around while they're running.
Kubernetes intro
Kubernetes (commonly referred to as "K8s") is an open-source system for automating deployment, scaling and management of containerized applications It aims to provide a "platform for automating deployment, scaling, and operations of application containers across clusters of hosts". We will see Kubernetes architecture, use cases, basics and live demo
Containerd internals: building a core container runtime
In this talk, we’ll briefly overview of the OpenWhisk serverless (function-as-a-service) framework that initially used the full Docker container engine as the execution vehicle for invoking user functions via containers. After several performance and stability challenges, this project decided to assess the various layers of the Docker engine (containerd and runC) as potential options for the function invoker. Out of that work came an open source project, bucketbench, that can be used to generate benchmarks of container lifecycle operations (e.g., start, stop, kill, remove, pause, unpause) and compare multithreaded operation throughput and stability of each optional engine.
This talk will provide details on the bucketbench project, explain how it has been used to generate performance data for these container runtimes, and shares lessons learned along the way that greatly impact container runtime performance, including bottlenecks in the Linux kernel.
In this talk you’ll learn how you can use bucketbench for your own performance tuning or assessment of container runtimes and how you can collaborate on improvements to the bucketbench project.
Migration of an Enterprise UI Microservice System from Cloud Foundry to Kuber...
Presented at Open Source Summit Japan with Jonathan Schweikhart on June 21, 2018.
Abstract: The 40 Node.js microservices making up the IBM Cloud UI historically have been deployed as apps on Cloud Foundry (CF), an open source PaaS. But, recently, this enterprise microservice system has been migrated to run on Kubernetes to take advantage of improved orchestration, higher availability, and better performance. Tony Erwin & Jonathan Schweikhart will discuss their team's journey and provide you with insights into the advantages of Kube over CF. Even more importantly, they will describe approaches to solving new problems that took the place of old ones, such as: 1) adapting PaaS apps to run as containers on Kube, 2) enabling geo load balancing between the different runtimes (to vette Kube before completely turning off CF), 3) integrating tools like Prometheus into existing monitoring systems, and more! Their team's first-hand experiences will help you avoid pitfalls as you prepare your own migrations to Kube!
Link to Info on Talk: https://ossalsjp18.sched.com/event/EaYj/migration-of-an-enterprise-ui-microservice-system-from-cloud-foundry-to-kubernetes-tony-erwin-jonathan-schweikhart-ibm?iframe=no
NOTE: CF is always evolving and the limitations on private networking and private host names mentioned in the slides are no longer current. If you have access to CF API 2.115.0 or higher (released on June 25, 2018), you can leverage CF's service discovery feature (see https://docs.cloudfoundry.org/devguide/deploy-apps/cf-networking.html#discovery ).
OpenEBS - Containerized Storage for Containers
These slides were presented in the containerization meet-up organized by digital ocean meet-up group in Bangalore. The slides talk about using containers for storage to make the storage truly non-disruptive during upgrades. This is a quick introduction to OpenEBS as well.
Swarm migration
This webinar will walk you through the steps involved in migrating a multi-container application deployed in Docker Swarm to Kubernetes. It will map the concepts of Swarm to Kubernetes. Attend this webinar to learn how to apply your Docker skills to Kubernetes for running and managing containerized applications in production.
Windows server and docker
Types of windows containers
Dockerize the .net application
Using container orchestration engine (Kubernetes) for .Net application
Kubernetes Webinar - Using ConfigMaps & Secrets
Many applications require configuration using some combination of configuration files, command line arguments, and environment variables. ConfigMaps in Kubernetes provide mechanisms to inject containers with configuration data while keeping them portable. Secrets decouple sensitive content from the pods using a volume plug-in. This webinar will discuss the use cases and scenarios for using ConfigMaps and Secrets.
Persistent Storage with Containers with Kubernetes & OpenShift
Manually configuring mounts for containers to various network storage platforms and services is tedious and time consuming. OpenShift and Kubernetes provides a rich library of volume plugins that allow authors of containerized applications (Pods) to declaratively specify what the storage requirements for the containers are so that OpenShift can dynamically provision and allocate the storage assets for the specified containers. As the author of the Kubernetes Persistent Volume specification, I will provide an overview of how Persistent Volume plugins work in OpenShift, demo block storage and file storage volume plugins and close with the Red Hat storage roadmap.
Presented at LinuxCon/ContainerCon by Mark Turansky, Principal Software Engineer, Red Hat
Mark Turansky is a Principal Software Engineer at Red Hat and a full-time contributor to the Kubernetes Project. Mark is the author of the Kubernetes Persistent Volume specification and a member of the Red Hat OpenShift Engineering team.
Kuberenetes - From Zero to Hero
Kubernetes has now become the de facto standard for deploying containerized applications at scale.
The presentation will follow K8s core concepts, architecture and real life scenarios.
Migrating our micro services from Java to Kotlin (Code.Talks 2018)
This talk elaborates our awesome journey migrate all our micro services from Java to Kotlin.
We will take a closer look at the challenges we faced as a team and the impact on our day to day work in an agile environment and how we overcame technical hurdles integrating with Spring Boot and introducing async workflows with Kotlin coroutines.
Finally we discuss at the actual benefits that we gained by moving all our micro services to Kotlin and do a critical review of our journey and an outlook at things to come.
Live Container Migration: OpenStack Summit Barcelona 2016
A talk presented by Phil Estes & Shaun Murakami, IBM Cloud Open Technologies, at the Barcelona OpenStack Summit on October 25, 2016. This talk covers a new feature that will be available in the Docker 1.13 engine for using the CRIU project to checkpoint and restore container processes on Linux. Phil & Shaun present details of this new capability and then demonstrate a proof-of-concept "live migration" of containers across nova compute hosts.
What's hot (20)
Aptira presents OpenStack Load Balancing as a Service at Banglore India OSUG ...
Aptira presents OpenStack Load Balancing as a Service at Banglore India OSUG ...
Bucketbench: Benchmarking Container Runtime Performance
Bucketbench: Benchmarking Container Runtime Performance
Serverless architecture: introduction & first steps
Serverless architecture: introduction & first steps
Containerd internals: building a core container runtime
Containerd internals: building a core container runtime
Migration of an Enterprise UI Microservice System from Cloud Foundry to Kuber...
Migration of an Enterprise UI Microservice System from Cloud Foundry to Kuber...
Persistent Storage with Containers with Kubernetes & OpenShift
Persistent Storage with Containers with Kubernetes & OpenShift
Migrating our micro services from Java to Kotlin (Code.Talks 2018)
Migrating our micro services from Java to Kotlin (Code.Talks 2018)
Live Container Migration: OpenStack Summit Barcelona 2016
Live Container Migration: OpenStack Summit Barcelona 2016
More from Natan Silnitsky
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Effective Strategies for Wix's Scaling challenges - GeeCon
This session unveils the multifaceted horizontal scaling strategies that power Wix's robust infrastructure. From Kafka consumer scaling and dynamic traffic routing for site segments to DynamoDB sharding and MySQL clusters custom routing with ProxySQL, we dissect the mechanisms that ensure scalability and performance at Wix.
Attendees will learn about the art of sharding and routing key selection across different systems, and how to apply these strategies to their own infrastructure. We'll share insights into choosing the right scaling strategy for various scenarios, balancing between managed services and custom solutions.
Key Takeaways:
- Grasp various sharding techniques and routing strategies used at Wix.
- Understand key considerations for sharding key and routing rule selection.
- Learn when and why to choose specific horizontal scaling strategies.
- Gain practical knowledge for applying these strategies to achieve scalability and high availability.
Join us to gain a blueprint for scaling your systems horizontally, drawing from Wix's proven practices.
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Discover how Wix transitioned from complex event sourcing and CQRS to streamlined CRUD services, optimizing their vast platform for better scalability, performance, and resiliency.
Wix's platform, designed to accommodate diverse business needs, boasts:
* 3.5 Billion daily HTTP transactions
* 70 Billion Kafka messages per day
* Roughly 4000 microservices in production
This session will highlight the simplification of Wix's architecture through domain events, resilient Kafka messaging, and advanced techniques like materialization and caching. By standardizing APIs and employing tools like protobuf and gRPC, Wix has enhanced the developer experience, both internally and externally, and fostered an open, integrative platform.
Attendees will gain insights into Wix's strategies for microservice coordination, ensuring system resilience and data consistency, as well as query performance optimization through innovative 2-level caching solutions.
Workflow Engines & Event Streaming Brokers - Can they work together? [Current...
Workflow engines and event streaming brokers offer very different solutions to the same requirement - an optimal implementation of microservices communication.
At Wix, we have a good experience with event-driven architecture for our 2500 microservices using Apache Kafka. Apache Kafka provides:
* support for very high throughput
* Fault tolerance
* very loose coupling
* Huge connectors eco-system
Temporal workflow orchestration has interesting features:
* Support for long running tasks
* business flows visual tracking
* Easy to follow imperative style programming
In this talk we will learn about the tradeoffs between the two technologies and how to implement various use cases in each architecture, including those that need a little more work.
DevSum - Lessons Learned from 2000 microservices
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
GeeCon - Lessons Learned from 2000 microservices
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Migrating to Multi Cluster Managed Kafka - ApacheKafkaIL
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Wix+Confluent Meetup - Lessons Learned from 2000 Event Driven Microservices
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
BuildStuff - Lessons Learned from 2000 Event Driven Microservices
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Lessons Learned from 2000 Event Driven Microservices - Reversim
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Devoxx Ukraine - Kafka based Global Data Mesh
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Devoxx UK - Migrating to Multi Cluster Managed Kafka
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Dev Days Europe - Kafka based Global Data Mesh at Wix
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Kafka Summit London - Kafka based Global Data Mesh at Wix
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
5 Takeaways from Migrating a Library to Scala 3 - Scala Love
Scala 3 is going to make Scala more easy to write, and especially read. more power features like enums and less misleading keywords like implicit.
But first we need to migrate our old Scala 2.12 / 2.13 codebase to Scala 3.
This talk tells the story of how I tried to migrate greyhound open-source library to Scala 3 with partial success.
You will hear about what works, what doesn't and about a few pitfalls to avoid
Migration takeaways include:
1. use migration tools, don't do it manually
2. which popular 3rd party libraries can and can't be used by Scala 3 code
and many more
Migrating to Multi Cluster Managed Kafka - DevopStars 2022
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Open sourcing a successful internal project - Reversim 2021
About a year ago data streams team at Wix has released to open-source its Kafka client SDK wrapper called Greyhound.
Greyhound offers rich functionality like message processing parallelisation and batching, various fault tolerant retry policies and much more.
This talk will show how the team designed Greyhound with a layered architecture to allow both public and private parts and also different levels of flexible configuration.
How it automatically syncs only relevant code from private repo to public one and also how it securely accepts public PRs back to the private repo.
Outline:
* Quick intro on what Greyhound is and its history at Wix
* Greyhound layered architecture design to allow both public and private parts and also different levels of flexible configuration.
* How it automatically syncs only relevant code from private repo to public one using Copybara tool
* how it securely accepts public PRs back to the private repo.
How to successfully manage a ZIO fiber’s lifecycle - Functional Scala 2021
Fibers are the backbone of the highly performant, asynchronous and concurrent abilities of ZIO. They are lightweight “green threads” implemented by the ZIO runtime system.
In this lightening talk you will learn about:
* How to handle fiber dying due to unexpected failure
* How to guarantee a ZIO fiber is interrupted
* How to set up fiber tracking and executing a fiberDump for increased debuggability
Advanced Caching Patterns used by 2000 microservices - Code Motion
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
More from Natan Silnitsky (20)
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Taming Distributed Systems: Key Insights from Wix's Large-Scale Experience - ...
Workflow Engines & Event Streaming Brokers - Can they work together? [Current...
Workflow Engines & Event Streaming Brokers - Can they work together? [Current...
Migrating to Multi Cluster Managed Kafka - ApacheKafkaIL
Migrating to Multi Cluster Managed Kafka - ApacheKafkaIL
Wix+Confluent Meetup - Lessons Learned from 2000 Event Driven Microservices
Wix+Confluent Meetup - Lessons Learned from 2000 Event Driven Microservices
BuildStuff - Lessons Learned from 2000 Event Driven Microservices
BuildStuff - Lessons Learned from 2000 Event Driven Microservices
Lessons Learned from 2000 Event Driven Microservices - Reversim
Lessons Learned from 2000 Event Driven Microservices - Reversim
Devoxx UK - Migrating to Multi Cluster Managed Kafka
Devoxx UK - Migrating to Multi Cluster Managed Kafka
Dev Days Europe - Kafka based Global Data Mesh at Wix
Dev Days Europe - Kafka based Global Data Mesh at Wix
Kafka Summit London - Kafka based Global Data Mesh at Wix
Kafka Summit London - Kafka based Global Data Mesh at Wix
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative
5 Takeaways from Migrating a Library to Scala 3 - Scala Love
5 Takeaways from Migrating a Library to Scala 3 - Scala Love
Migrating to Multi Cluster Managed Kafka - DevopStars 2022
Migrating to Multi Cluster Managed Kafka - DevopStars 2022
Open sourcing a successful internal project - Reversim 2021
Open sourcing a successful internal project - Reversim 2021
How to successfully manage a ZIO fiber’s lifecycle - Functional Scala 2021
How to successfully manage a ZIO fiber’s lifecycle - Functional Scala 2021
Advanced Caching Patterns used by 2000 microservices - Code Motion
Advanced Caching Patterns used by 2000 microservices - Code Motion
Recently uploaded
Epistemic Interaction - tuning interfaces to provide information for AI support
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
When stars align: studies in data quality, knowledge graphs, and machine lear...
Keynote at DQMLKG workshop at the 21st European Semantic Web Conference 2024
Bits & Pixels using AI for Good.........
A whirlwind tour of tech & AI for socio-environmental impact.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Elevating Tactical DDD Patterns Through Object Calisthenics
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Presented by Suzanne Phillips and Alex Marcotte
The Art of the Pitch: WordPress Relationships and Sales
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI)
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Reflecting on new architectures for knowledge based systems in light of generative ai
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
“AGI should be open source and in the public domain at the service of humanity and the planet.”
UiPath Test Automation using UiPath Test Suite series, part 4
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase Enterprise-Wide Efficiency by VP Data Platform, American Express
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head of Product, Amazon Games
Key Trends Shaping the Future of Infrastructure.pdf
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovation With Your Product by VP of Product Design, Warner Music Group
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP of Product, Amplitude
Recently uploaded (20)
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf
When stars align: studies in data quality, knowledge graphs, and machine lear...
When stars align: studies in data quality, knowledge graphs, and machine lear...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
The Art of the Pitch: WordPress Relationships and Sales
The Art of the Pitch: WordPress Relationships and Sales
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
To Graph or Not to Graph Knowledge Graph Architectures and LLMs
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
UiPath Test Automation using UiPath Test Suite series, part 4
UiPath Test Automation using UiPath Test Suite series, part 4
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
From Daily Decisions to Bottom Line: Connecting Product Work to Revenue by VP...
Wix meetup Feb 26 2020 - How to build 1000 microservices with kafka and thrive
- 3. * Wix Scale / concerns
- 9. * polyglot
- 10. REST
- 11. * Easily change
- 21. * broker -> Consumer
- 26. * No explicit commit
- 28. REST
- 29. message HtmlSitePublished { int64 revision = 1 ; // Revision string site_id = 2; } REST
- 52. Ordered
- 53. Unordered
- 59. * Each service, metrics like
- 61. * First Group topic
- 69. * In memory
- 70. DC A (Kafka Cluster A) DC B (Kafka Cluster A)
- 71. DC B (Kafka Cluster A) DC A (Kafka Cluster A) COPIES IN REAL TIME
- 74. Raven
- 75. Raven