Scala 3 is going to make Scala more easy to write, and especially read. more power features like enums and less misleading keywords like implicit.
But first we need to migrate our old Scala 2.12 / 2.13 codebase to Scala 3.
This talk tells the story of how I tried to migrate greyhound open-source library to Scala 3 with partial success.
You will hear about what works, what doesn't and about a few pitfalls to avoid
Migration takeaways include:
1. use migration tools, don't do it manually
2. which popular 3rd party libraries can and can't be used by Scala 3 code
and many more
While Go is the language-of-choice in the cloud-native world, Python has a huge community and makes it really easy to extend Kubernetes in only a few lines of code.
This talk shows examples on how to use Python to query the Kubernetes API, how to write simple controllers in only 10 lines of Python, how to build complete web UIs, and how to test everything with py.test and Kind.
Some of the open-source projects which will be covered: pykube-ng, Kubernetes Web View, kube-janitor, and Kopf (Kubernetes Operator Pythonic Framework).
Talk held in Prague on 2019-09-05:
https://www.meetup.com/Cloud-Native-Prague/events/263802447/
Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics.
Presentation by Colin McCabe, Confluent, Big Data Day LA
Dynamic Components using Single-Page-Application Concepts in AEM/CQNetcetera
Dynamic components display content dependable on context, hence they cannot be cached.
Out of the box, Adobe Experience Manager doesn't give us many options for granular caching on a component level.
When faced with this problem, we usually resort to developing components that are leveraging Server-Side Includes or AJAX to get the HTML with dynamic data.
As an alternative solution, we have also developed dynamic components that use Single Page Application concepts, by using templates and JSON-responses, to provide the same dynamic behavior.
In this presentation we will cover all of the dynamic components types, compare the benefits and drawbacks of each, and state the use-cases where each can be effectively applied.
We will take a deeper look at the dynamic components done with SPA concepts, as they are rarity in the AEM world, and also provide a walk-through of the technologies used, how some common problems were solved, as well as the benefits that have been gained by their usage.
While Go is the language-of-choice in the cloud-native world, Python has a huge community and makes it really easy to extend Kubernetes in only a few lines of code.
This talk shows examples on how to use Python to query the Kubernetes API, how to write simple controllers in only 10 lines of Python, how to build complete web UIs, and how to test everything with py.test and Kind.
Some of the open-source projects which will be covered: pykube-ng, Kubernetes Web View, kube-janitor, and Kopf (Kubernetes Operator Pythonic Framework).
Talk held in Prague on 2019-09-05:
https://www.meetup.com/Cloud-Native-Prague/events/263802447/
Building distributed systems is challenging. Luckily, Apache Kafka provides a powerful toolkit for putting together big services as a set of scalable, decoupled components. In this talk, I'll describe some of the design tradeoffs when building microservices, and how Kafka's powerful abstractions can help. I'll also talk a little bit about what the community has been up to with Kafka Streams, Kafka Connect, and exactly-once semantics.
Presentation by Colin McCabe, Confluent, Big Data Day LA
Dynamic Components using Single-Page-Application Concepts in AEM/CQNetcetera
Dynamic components display content dependable on context, hence they cannot be cached.
Out of the box, Adobe Experience Manager doesn't give us many options for granular caching on a component level.
When faced with this problem, we usually resort to developing components that are leveraging Server-Side Includes or AJAX to get the HTML with dynamic data.
As an alternative solution, we have also developed dynamic components that use Single Page Application concepts, by using templates and JSON-responses, to provide the same dynamic behavior.
In this presentation we will cover all of the dynamic components types, compare the benefits and drawbacks of each, and state the use-cases where each can be effectively applied.
We will take a deeper look at the dynamic components done with SPA concepts, as they are rarity in the AEM world, and also provide a walk-through of the technologies used, how some common problems were solved, as well as the benefits that have been gained by their usage.
Kubernetes has two simple but powerful network concepts: every Pod is connected to the same network, and Services let you talk to a Pod by name. Bryan will take you through how these concepts are implemented - Pod Networks via the Container Network Interface (CNI), Service Discovery via kube-dns and Service virtual IPs, then on to how Services are exposed to the rest of the world.
Performance Troubleshooting Using Apache Spark MetricsDatabricks
Performance troubleshooting of distributed data processing systems is a complex task. Apache Spark comes to rescue with a large set of metrics and instrumentation that you can use to understand and improve the performance of your Spark-based applications. You will learn about the available metric-based instrumentation in Apache Spark: executor task metrics and the Dropwizard-based metrics system. The talk will cover how Hadoop and Spark service at CERN is using Apache Spark metrics for troubleshooting performance and measuring production workloads. Notably, the talk will cover how to deploy a performance dashboard for Spark workloads and will cover the use of sparkMeasure, a tool based on the Spark Listener interface. The speaker will discuss the lessons learned so far and what improvements you can expect in this area in Apache Spark 3.0.
DoK Talks #91- Leveraging Druid Operator to manage Apache Druid on KubernetesDoKC
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
This talk will introduce Druid-Operator (https://github.com/druid-io/druid-operator) and how Kubernetes and Operator framework can be used to write an operator that enables provisioning, management, and scaling of a complex cluster of Apache Druid to 1000s of nodes.
This talk will cover why Kubernetes and the operator framework is a perfect fit for managing a complex stateful application. Learnings and pitfalls faced while writing the Druid-Operator and the special considerations to be taken into account when managing and scaling a stateful application.
BIO
Adheip Singh is a software engineer at Rill Data Inc. He is maintainer of druid-operator and kubectl-druid-plugin. Active contributor the operator ecosystem.
TALK TAKEAWAYS
1.This session will help anyone looking for running Apache Druid on Kubernetes
2.It will help understand how someone can write operators to simplify the management of complex applications
3.Special considerations to take into account when managing and scaling a stateful application.
4.How to scope an operator within multiple namespaces
5.Druid operator features and how it scales and manages the underlying statefulsets and deployments.
( ELK Stack Training - https://www.edureka.co/elk-stack-trai... )
This Kibana tutorial by Edureka will give you an introduction to the Kibana 5 Dashboard and help you get started with working on the ELK Stack. Below are the topics covered in this Kibana tutorial video:
1. Introduction To ELK Stack
2. Role Of Kibana In ELK
3. Kibana 5 Dashboard
4. Demo: Kibana For Visualization & Analytics
Tuning Apache Kafka Connectors for Flink.pptxFlink Forward
Flink Forward San Francisco 2022.
In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you!
by
Olena Babenko
This talk discusses the core concepts behind the Kubernetes extensibility model. We are going to see how to implement new CRDs, operators and when to use them to automate the most critical aspects of your Kubernetes clusters.
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...Databricks
Apache Spark is a wonderful platform for running your analytics jobs. It has great ingestion features from CSV, Hive, JDBC, etc. however, you may have your own data sources or formats you want to use. Your solution could be to convert your data in a CSV or JSON file and then ask Spark to do ingest it through its built-in tools. However, for enhanced performance, we will explore the way to build a data source, in Java, to extend Spark’s ingestion capabilities. We will first understand how Spark works for ingestion, then walk through the development of this data source plug-in. Targeted audience Software and data engineers who need to expand Spark’s ingestion capability. Key takeaways Requirements, needs & architecture – 15%. Build the required tool set in Java – 85%.
Kafka Tutorial - introduction to the Kafka streaming platformJean-Paul Azar
Why is Kafka so fast? Why is Kafka so popular? Why Kafka?
Introduction to Kafka streaming platform. Covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example. Lastly, we added some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have started to expand on the Java examples to correlate with the design discussion of Kafka. We have also expanded on the Kafka design section and added references.
Google jib: Building Java containers without DockerMaarten Smeets
In this quick introduction to Google Jib I'll show what it solves, how you can use it and how it compares to some other solutions. Also see https://www.youtube.com/watch?v=vl-U9m8EXT8
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
Flink Forward San Francisco 2022.
This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications.
We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs.
After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are.
by
David Moravek
Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.
Common issues with Apache Kafka® Producerconfluent
Badai Aqrandista, Confluent, Senior Technical Support Engineer
This session will be about a common issue in the Kafka Producer: producer batch expiry. We will be discussing the Kafka Producer internals, its common causes, such as a slow network or small batching, and how to overcome them. We will also be sharing some examples along the way!
https://www.meetup.com/apache-kafka-sydney/events/279651982/
Real-time streaming and data pipelines with Apache KafkaJoe Stein
Get up and running quickly with Apache Kafka http://kafka.apache.org/
* Fast * A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
* Scalable * Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers
* Durable * Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.
* Distributed by Design * Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Kubernetes has two simple but powerful network concepts: every Pod is connected to the same network, and Services let you talk to a Pod by name. Bryan will take you through how these concepts are implemented - Pod Networks via the Container Network Interface (CNI), Service Discovery via kube-dns and Service virtual IPs, then on to how Services are exposed to the rest of the world.
Performance Troubleshooting Using Apache Spark MetricsDatabricks
Performance troubleshooting of distributed data processing systems is a complex task. Apache Spark comes to rescue with a large set of metrics and instrumentation that you can use to understand and improve the performance of your Spark-based applications. You will learn about the available metric-based instrumentation in Apache Spark: executor task metrics and the Dropwizard-based metrics system. The talk will cover how Hadoop and Spark service at CERN is using Apache Spark metrics for troubleshooting performance and measuring production workloads. Notably, the talk will cover how to deploy a performance dashboard for Spark workloads and will cover the use of sparkMeasure, a tool based on the Spark Listener interface. The speaker will discuss the lessons learned so far and what improvements you can expect in this area in Apache Spark 3.0.
DoK Talks #91- Leveraging Druid Operator to manage Apache Druid on KubernetesDoKC
https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
This talk will introduce Druid-Operator (https://github.com/druid-io/druid-operator) and how Kubernetes and Operator framework can be used to write an operator that enables provisioning, management, and scaling of a complex cluster of Apache Druid to 1000s of nodes.
This talk will cover why Kubernetes and the operator framework is a perfect fit for managing a complex stateful application. Learnings and pitfalls faced while writing the Druid-Operator and the special considerations to be taken into account when managing and scaling a stateful application.
BIO
Adheip Singh is a software engineer at Rill Data Inc. He is maintainer of druid-operator and kubectl-druid-plugin. Active contributor the operator ecosystem.
TALK TAKEAWAYS
1.This session will help anyone looking for running Apache Druid on Kubernetes
2.It will help understand how someone can write operators to simplify the management of complex applications
3.Special considerations to take into account when managing and scaling a stateful application.
4.How to scope an operator within multiple namespaces
5.Druid operator features and how it scales and manages the underlying statefulsets and deployments.
( ELK Stack Training - https://www.edureka.co/elk-stack-trai... )
This Kibana tutorial by Edureka will give you an introduction to the Kibana 5 Dashboard and help you get started with working on the ELK Stack. Below are the topics covered in this Kibana tutorial video:
1. Introduction To ELK Stack
2. Role Of Kibana In ELK
3. Kibana 5 Dashboard
4. Demo: Kibana For Visualization & Analytics
Tuning Apache Kafka Connectors for Flink.pptxFlink Forward
Flink Forward San Francisco 2022.
In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you!
by
Olena Babenko
This talk discusses the core concepts behind the Kubernetes extensibility model. We are going to see how to implement new CRDs, operators and when to use them to automate the most critical aspects of your Kubernetes clusters.
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...Databricks
Apache Spark is a wonderful platform for running your analytics jobs. It has great ingestion features from CSV, Hive, JDBC, etc. however, you may have your own data sources or formats you want to use. Your solution could be to convert your data in a CSV or JSON file and then ask Spark to do ingest it through its built-in tools. However, for enhanced performance, we will explore the way to build a data source, in Java, to extend Spark’s ingestion capabilities. We will first understand how Spark works for ingestion, then walk through the development of this data source plug-in. Targeted audience Software and data engineers who need to expand Spark’s ingestion capability. Key takeaways Requirements, needs & architecture – 15%. Build the required tool set in Java – 85%.
Kafka Tutorial - introduction to the Kafka streaming platformJean-Paul Azar
Why is Kafka so fast? Why is Kafka so popular? Why Kafka?
Introduction to Kafka streaming platform. Covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example. Lastly, we added some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have started to expand on the Java examples to correlate with the design discussion of Kafka. We have also expanded on the Kafka design section and added references.
Google jib: Building Java containers without DockerMaarten Smeets
In this quick introduction to Google Jib I'll show what it solves, how you can use it and how it compares to some other solutions. Also see https://www.youtube.com/watch?v=vl-U9m8EXT8
Slides for the webinar held on January 21st 2014
Repair & Recovery for your MySQL, MariaDB & MongoDB / TokuMX Clusters
Galera Cluster, NDB Cluster, VIP with HAProxy and Keepalived, MongoDB Sharded Cluster, etc. all have their own availability models. We are aware of these availability models and will demonstrate in this webinar how to take corrective action in case of failures via our cluster management tool, ClusterControl.
In this webinar, Severalnines CTO Johan Andersson will show you how to leverage ClusterControl to detect failures in your database cluster and automatically repair them to maximize the availability of your database services. And Codership CEO Seppo Jaakola will be joining Johan to provide a deep-dive into Galera recovery internals.
Agenda:
Redundancy models for Galera, NDB and MongoDB/TokuMX
Failover & Recovery (Automatic vs Manual)
Zooming into Galera recovery procedures
Split brains in multi-datacenter setups
Flink Forward San Francisco 2022.
This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications.
We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs.
After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are.
by
David Moravek
Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.
Common issues with Apache Kafka® Producerconfluent
Badai Aqrandista, Confluent, Senior Technical Support Engineer
This session will be about a common issue in the Kafka Producer: producer batch expiry. We will be discussing the Kafka Producer internals, its common causes, such as a slow network or small batching, and how to overcome them. We will also be sharing some examples along the way!
https://www.meetup.com/apache-kafka-sydney/events/279651982/
Real-time streaming and data pipelines with Apache KafkaJoe Stein
Get up and running quickly with Apache Kafka http://kafka.apache.org/
* Fast * A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.
* Scalable * Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers
* Durable * Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.
* Distributed by Design * Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
Boost your productivity with Scala tooling!MeriamLachkar1
Our rich ecosystem provides developers with powerful tools that improve productivity on small or huge projects.
In this talk, I will present the tools that allow me to focus on my projects by making tedious tasks easier. From bootstrapping projects, to code linting and refactoring, from continuous integration and automatic publication and documentation rendering, come discover my favorite tools.
Take a look at what Rails 5 has in store for you. We go through all the new features and improvements across development, testing, caching and much more. So let's dive in.
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...Shirshanka Das
This talk describes the motivations behind Apache Gobblin (incubating), architecture, latest innovations in supporting both batch and streaming data pipelines as well as future roadmap.
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Zabbix
At DBC we are running docker and other container types in a mesos/marathon cluster environment. I will demonstrate how we collect statistics, logs etc. and monitor this environment, showing configuration examples, data flows and templates.
Some of the covered topics:
- Mesos master and agents
- Marathon Framework
- Docker engine
- Containers
- Zookeeper
- Elasticserach/ELK
This presentation gives a pragmatic introduction to SBT - the Scala Build Tool - in 7 examples. Each example is a build in it's own root directory: ./example?? Beginning with very simple sbt examples the later examples are becoming more structured and more complex showing multi-project builds, cross version builds, packaging and publishing, custom Settings and Tasks and the integration of the Ammonite REPL into your build. We also look at InputTasks, Commands and plugin development.
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Effective Strategies for Wix's Scaling challenges - GeeConNatan Silnitsky
This session unveils the multifaceted horizontal scaling strategies that power Wix's robust infrastructure. From Kafka consumer scaling and dynamic traffic routing for site segments to DynamoDB sharding and MySQL clusters custom routing with ProxySQL, we dissect the mechanisms that ensure scalability and performance at Wix.
Attendees will learn about the art of sharding and routing key selection across different systems, and how to apply these strategies to their own infrastructure. We'll share insights into choosing the right scaling strategy for various scenarios, balancing between managed services and custom solutions.
Key Takeaways:
- Grasp various sharding techniques and routing strategies used at Wix.
- Understand key considerations for sharding key and routing rule selection.
- Learn when and why to choose specific horizontal scaling strategies.
- Gain practical knowledge for applying these strategies to achieve scalability and high availability.
Join us to gain a blueprint for scaling your systems horizontally, drawing from Wix's proven practices.
Discover how Wix transitioned from complex event sourcing and CQRS to streamlined CRUD services, optimizing their vast platform for better scalability, performance, and resiliency.
Wix's platform, designed to accommodate diverse business needs, boasts:
* 3.5 Billion daily HTTP transactions
* 70 Billion Kafka messages per day
* Roughly 4000 microservices in production
This session will highlight the simplification of Wix's architecture through domain events, resilient Kafka messaging, and advanced techniques like materialization and caching. By standardizing APIs and employing tools like protobuf and gRPC, Wix has enhanced the developer experience, both internally and externally, and fostered an open, integrative platform.
Attendees will gain insights into Wix's strategies for microservice coordination, ensuring system resilience and data consistency, as well as query performance optimization through innovative 2-level caching solutions.
Workflow Engines & Event Streaming Brokers - Can they work together? [Current...Natan Silnitsky
Workflow engines and event streaming brokers offer very different solutions to the same requirement - an optimal implementation of microservices communication.
At Wix, we have a good experience with event-driven architecture for our 2500 microservices using Apache Kafka. Apache Kafka provides:
* support for very high throughput
* Fault tolerance
* very loose coupling
* Huge connectors eco-system
Temporal workflow orchestration has interesting features:
* Support for long running tasks
* business flows visual tracking
* Easy to follow imperative style programming
In this talk we will learn about the tradeoffs between the two technologies and how to implement various use cases in each architecture, including those that need a little more work.
DevSum - Lessons Learned from 2000 microservicesNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
GeeCon - Lessons Learned from 2000 microservicesNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Migrating to Multi Cluster Managed Kafka - ApacheKafkaILNatan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
BuildStuff - Lessons Learned from 2000 Event Driven MicroservicesNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Lessons Learned from 2000 Event Driven Microservices - ReversimNatan Silnitsky
Wix has a huge scale of event driven traffic. More than 70 billion Kafka business events per day.
Over the past few years Wix has made a gradual transition to an event-driven architecture for its 2000 microservices.
We have made mistakes along the way but have improved and learned a lot about how to make sure our production is still maintainable, performant and resilient.
In this talk you will hear about the lessons we learned including:
1. The importance of atomic operations for databases and events
2. avoiding data consistency issues due to out-of-order and duplicate processing
3. Having essential events debugging and quick-fix tools in production
and a few more
Devoxx Ukraine - Kafka based Global Data MeshNatan Silnitsky
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Devoxx UK - Migrating to Multi Cluster Managed KafkaNatan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Dev Days Europe - Kafka based Global Data Mesh at WixNatan Silnitsky
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Kafka Summit London - Kafka based Global Data Mesh at WixNatan Silnitsky
As your organization rapidly grows in scale, so do the amount of challenges.
Growing scale comes in multiple dimensions - traffic, geographic presence, products portfolio, various technologies, amount of developers, etc.
Coming up with an architecture that can handle all of the data flows in a universal, simple way is key.
This talk is about Wix's Kafka based global data architecture and platform.
How we made it very easy for Wix 2000 microservices to publish and subscribe to data, no matter where they are deployed in the world, or what technological stack they use.
All the while offering various SDKs (some of them open-source), tools, and features for adapting to growing scale and insuring high resilience.
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative Natan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Migrating to Multi Cluster Managed Kafka - DevopStars 2022Natan Silnitsky
As Wix Kafka usage grew to 2.5B messages per day, >20K topics and >100K leader partitions serving 2000 microservices,
we decided to migrate from self-operated single cluster per data-center to a managed cloud service (Like Amazon MSK or Confluent Cloud) with a multi-cluster setup.
The classic approach would be to perform this transition when all incoming traffic is removed from the data center.
But draining an entire data-center for an undetermined period of time, until all 2000 services complete the switch was too risky for us.
This talk is about how we gradually migrated all of our Kafka consumers and producers with 0 downtime while they continued to handle regular traffic. You will learn practical steps you can take to greatly reduce the risks and speed up the migration timeline.
Open sourcing a successful internal project - Reversim 2021Natan Silnitsky
About a year ago data streams team at Wix has released to open-source its Kafka client SDK wrapper called Greyhound.
Greyhound offers rich functionality like message processing parallelisation and batching, various fault tolerant retry policies and much more.
This talk will show how the team designed Greyhound with a layered architecture to allow both public and private parts and also different levels of flexible configuration.
How it automatically syncs only relevant code from private repo to public one and also how it securely accepts public PRs back to the private repo.
Outline:
* Quick intro on what Greyhound is and its history at Wix
* Greyhound layered architecture design to allow both public and private parts and also different levels of flexible configuration.
* How it automatically syncs only relevant code from private repo to public one using Copybara tool
* how it securely accepts public PRs back to the private repo.
How to successfully manage a ZIO fiber’s lifecycle - Functional Scala 2021Natan Silnitsky
Fibers are the backbone of the highly performant, asynchronous and concurrent abilities of ZIO. They are lightweight “green threads” implemented by the ZIO runtime system.
In this lightening talk you will learn about:
* How to handle fiber dying due to unexpected failure
* How to guarantee a ZIO fiber is interrupted
* How to set up fiber tracking and executing a fiberDump for increased debuggability
Advanced Caching Patterns used by 2000 microservices - Code MotionNatan Silnitsky
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
Advanced Caching Patterns used by 2000 microservices - Devoxx UkraineNatan Silnitsky
Wix has a huge scale of traffic. more than 500 billion HTTP requests and more than 1.5 billion Kafka business events per day. This talk goes through 3 Caching Patterns that are used by Wix's 2000 microservices in order to provide the best experience for Wix users along with saving costs and increasing availability.
A cache will reduce latency, by avoiding the need of a costly query to a DB, a HTTP request to a Wix servicer, or a 3rd-party service. It will reduce the needed scale to service these costly requests. It will also improve reliability, by making sure some data can be returned even if aforementioned DB or 3rd-party service are currently unavailable.
The patterns include:
* Configuration Data Cache - persisted locally or to S3
* HTTP Reverse Proxy Caching - using Varnish Cache
* (Dynamo)DB+CDC based Cache and more - for unlimited capacity with continuously updating LRU cache on top each pattern is optimal for other use cases, but all allow to reduce costs and gain performance and resilience.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
A Comprehensive Look at Generative AI in Retail App Testing.pdfkalichargn70th171
Traditional software testing methods are being challenged in retail, where customer expectations and technological advancements continually shape the landscape. Enter generative AI—a transformative subset of artificial intelligence technologies poised to revolutionize software testing.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
2. @NSilnitsky
Migrating a library to Scala 3
Greyhound
A Scala/Java high-level SDK
for Apache Kafka.
Powered by ZIO
Takeaways from
migrating a library
5. SBT
Migrating a library to Scala 3
Scala 2.12
Bazel
Scala
2.13
Scala
3.1
Migrating Greyhound library
6. Migrating a library to Scala 3
SBT
Scala 2.12
Bazel
Scala
2.13
Scala
3.1
Migrating Greyhound library
Step 1:
Make SBT Work
7. Migrating a library to Scala 3
Bazel does not support
Scala 3 yet.
And SBT has…
→ a Nice Migration Plugin
→ dedicated syntax
SBT
Scala 2.12
Bazel
Why?
Step 1:
Make SBT Work
23. @NSilnitsky
@NSilnitsky
val offsets = records.groupBy(RecordTopicPartition(_)).mapValues(_.maxBy(_.offset).offset + 1)
val offsets = records.groupBy(RecordTopicPartition(_)).mapValues(...).toMap
> scalafix Collection213Roughly
[info] Running scalafix on 69 Scala sources
[success] Total time: 24 s, completed 16 Feb 2022, 22:22:18
Ended up fixing it manually:
24. Migrating a library to Scala 3
SBT
Scala 2.12
Bazel
Scala
2.13
Scala
3.1
Step 1:
Make SBT Work
Step 2:
Switch to Scala 2.13
Step 3:
Switch to Scala 3
25. from
Scala 2.13
to
Scala 3 Scala 3
compiler
Scala 2.13
compiler
3rd-party
library dep
Scala 2.13 TASTy Reader can read TASTy format
Scala3 Unpickler
Classpath
Interoperability
can read Pickle format
26. Just use the migration tool
from
Scala 2.13
to
Scala 3
SBT migration
plugin
27. @NSilnitsky
helps you migrate your build & code
to Scala3.
Use SBT Migration Plugin.
Migrating a library to Scala 3
👉 addSbtPlugin("ch.epfl.scala" % "sbt-scala3-migrate" % "0.5.0")
migrate-libs
migrate-scalacOptions
migrate-syntax
migrate
31. @NSilnitsky
> migrate-syntax root
Migrating a library to Scala 3
These changes are not mandatory, and caused many new 2.13 compiler
errors → Rolled back
[error] /…/src/main/scala/future-interop/com/wixpress/dst/greyhound/future/GreyhoundRuntime.scala:27:19: type mismatch;
[error] found : zio.clock.Clock.Service.live.type (with underlying type zio.clock.Clock.Service)
[error] required: zio.blocking.Blocking.Service with zio.random.Random.Service with zio.system.System.Service with
zio.console.Console.Service with zio.clock.Clock.Service
[error] Clock.Service.live,
[error] ^
val zenv: Has[Clock.Service] with Has[Console.Service] with Has[System.Service] with Has[Random.Service] with
Has[Blocking.Service]
32. compiles in scala 3 with options:
> migrate root
Migrating a library to Scala 3
-source:3.0-migration
-rewrite
34. @NSilnitsky
@NSilnitsky
rewrite mode:
Once your code compiles in the migration mode,
Warnings are resolved by the compiler itself
scalacOptions ++= Seq(
"-source:3.0-migration", “-rewrite”
),
➔ Rewrites are NOT applied if the code compiles in error.
➔ You cannot choose which rules are applied (compiler runs all of them).
Scala compiler options
35. @NSilnitsky
> migrate root
Migrating a library to Scala 3
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.scalameta:semanticdb-scalac_2.13.8:4.4.31
[error] Not found
[error] Not found
[error] not found: /Users/natans/.ivy2/localorg.scalameta/semanticdb-scalac_2.13.8/4.4.31/ivys/ivy.xml
[error] not found:
https://repo1.maven.org/maven2/org/scalameta/semanticdb-scalac_2.13.8/4.4.31/semanticdb-scalac_2.13.8-4.4.31.pom
36. @NSilnitsky
> migrate root
Migrating a library to Scala 3
[warn]
[warn] Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading org.scalameta:semanticdb-scalac_2.13.8:4.4.31
[error] Not found
[error] Not found
[error] not found: /Users/natans/.ivy2/localorg.scalameta/semanticdb-scalac_2.13.8/4.4.31/ivys/ivy.xml
[error] not found:
https://repo1.maven.org/maven2/org/scalameta/semanticdb-scalac_2.13.8/4.4.31/semanticdb-scalac_2.13.8-4.4.31.pom
Failed for 2.13.8
41. @NSilnitsky
5 other build tools for support
➔ Bazel - work in progress - ETA - end of Q1
➔ Gradle - Scala 3 supported version 7.3
➔ Mill - support in 0.9.x
➔ Maven - 4.5.x (scala-maven-plugin)
➔ Native Cross compilation for your library
Migrating a library to Scala 3