Why Reactive Architecture Will Take Over The World (and why we should be wary...Steve Pember
The natural tendency for application developers is to construct their code in a procedural, monolithic pattern. Veteran Developers know that this leads to error prone, unscalable, slow software – yet it is alarmingly prevalent. There have been several architectural patterns that have risen over the years which have attempted to mitigate this problem. We’ve heard of Service Oriented Architecture, Integration Patterns, and Event-Driven Systems, but the Reactive pattern has the best chance for success.
In this talk I will discuss the tenants of the Reactive Pattern and the importance of moving away from Monolithic architectures into Reactive. We will examine Spring Integration and the Grails Async features (along with Netty and RabbitMQ) in order to show they can quickly and effectively help your application to become Reactive. Finally, I will argue that the JVM is the best foundation currently for this architecture – but that if we’re not careful, NodeJS may be the most popular.
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsJonas Bonér
The demands and expectations for applications have changed dramatically in recent years. Applications today are deployed on a wide range of infrastructure; from mobile devices up to thousands of nodes running in the cloud—all powered by multi-core processors. They need to be rich and collaborative, have a real-time feel with millisecond response time and should never stop running. Additionally, modern applications are a mashup of external services that need to be consumed and composed to provide the features at hand.
We are seeing a new type of applications emerging to address these new challenges—these are being called Reactive Applications. In this talk we will discuss four key traits of Reactive; Event-Driven, Scalable, Resilient and Responsive—how they impact application design, how they interact, their supporting technologies and techniques, how to think when designing and building them—all to make it easier for you and your team to Go Reactive.
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsJonas Bonér
Abstract:
The demands and expectations for applications have changed dramatically in recent years. Applications today are deployed on a wide range of infrastructure; from mobile devices up to thousands of nodes running in the cloud—all powered by multi-core processors. They need to be rich and collaborative, have a real-time feel with millisecond response time and should never stop running. Additionally, modern applications are a mashup of external services that need to be consumed and composed to provide the features at hand.
We are seeing a new type of applications emerging to address these new challenges—these are being called Reactive Applications. In this talk we will discuss four key traits of Reactive; Responsive, Resilient, Elastic and Message-Driven—how they impact application design, how they interact, their supporting technologies and techniques, how to think when designing and building them—all to make it easier for you and your team to Go Reactive.
Intended Audience:
Programmers, architects, CIO/CTOs and everyone with a desire to challenge the status quo and expand their horizons on how to tackle the current and future challenges in the computing industry.
Reactive Design Patterns — J on the BeachRoland Kuhn
Our software needs to become reactive, this realization is widely understood: we need to consider responsiveness, maintainability, elasticity and scalability from the outset. Not all systems need to implement all these to the same degree, specific project requirements will determine where effort is most wisely spent, but in the vast majority of cases the need to go reactive will demand that we design our applications differently.
In this presentation we explore several architecture elements that are commonly found in reactive systems (like the circuit breaker, various replication techniques, or flow control protocols). These patterns are language agnostic and also independent of the abundant choice of reactive programming frameworks and libraries, they are well-specified starting points for exploring the design space of a concrete problem: thinking is strictly required!
Reactive Streams: Handling Data-Flow the Reactive WayRoland Kuhn
Building on the success of Reactive Extensions—first in Rx.NET and now in RxJava—we are taking Observers and Observables to the next level: by adding the capability of handling back-pressure between asynchronous execution stages we enable the distribution of stream processing across a cluster of potentially thousands of nodes. The project defines the common interfaces for interoperable stream implementations on the JVM and is the result of a collaboration between Twitter, Netflix, Pivotal, RedHat and Typesafe. In this presentation I introduce the guiding principles behind its design and show examples using the actor-based implementation in Akka.
Reactive Programming, Traits and Principles. What is Reactive, where does it come from, and what is it good for? How does it differ from event driven programming? It only functional?
Why Reactive Architecture Will Take Over The World (and why we should be wary...Steve Pember
The natural tendency for application developers is to construct their code in a procedural, monolithic pattern. Veteran Developers know that this leads to error prone, unscalable, slow software – yet it is alarmingly prevalent. There have been several architectural patterns that have risen over the years which have attempted to mitigate this problem. We’ve heard of Service Oriented Architecture, Integration Patterns, and Event-Driven Systems, but the Reactive pattern has the best chance for success.
In this talk I will discuss the tenants of the Reactive Pattern and the importance of moving away from Monolithic architectures into Reactive. We will examine Spring Integration and the Grails Async features (along with Netty and RabbitMQ) in order to show they can quickly and effectively help your application to become Reactive. Finally, I will argue that the JVM is the best foundation currently for this architecture – but that if we’re not careful, NodeJS may be the most popular.
Go Reactive: Event-Driven, Scalable, Resilient & Responsive SystemsJonas Bonér
The demands and expectations for applications have changed dramatically in recent years. Applications today are deployed on a wide range of infrastructure; from mobile devices up to thousands of nodes running in the cloud—all powered by multi-core processors. They need to be rich and collaborative, have a real-time feel with millisecond response time and should never stop running. Additionally, modern applications are a mashup of external services that need to be consumed and composed to provide the features at hand.
We are seeing a new type of applications emerging to address these new challenges—these are being called Reactive Applications. In this talk we will discuss four key traits of Reactive; Event-Driven, Scalable, Resilient and Responsive—how they impact application design, how they interact, their supporting technologies and techniques, how to think when designing and building them—all to make it easier for you and your team to Go Reactive.
Go Reactive: Building Responsive, Resilient, Elastic & Message-Driven SystemsJonas Bonér
Abstract:
The demands and expectations for applications have changed dramatically in recent years. Applications today are deployed on a wide range of infrastructure; from mobile devices up to thousands of nodes running in the cloud—all powered by multi-core processors. They need to be rich and collaborative, have a real-time feel with millisecond response time and should never stop running. Additionally, modern applications are a mashup of external services that need to be consumed and composed to provide the features at hand.
We are seeing a new type of applications emerging to address these new challenges—these are being called Reactive Applications. In this talk we will discuss four key traits of Reactive; Responsive, Resilient, Elastic and Message-Driven—how they impact application design, how they interact, their supporting technologies and techniques, how to think when designing and building them—all to make it easier for you and your team to Go Reactive.
Intended Audience:
Programmers, architects, CIO/CTOs and everyone with a desire to challenge the status quo and expand their horizons on how to tackle the current and future challenges in the computing industry.
Reactive Design Patterns — J on the BeachRoland Kuhn
Our software needs to become reactive, this realization is widely understood: we need to consider responsiveness, maintainability, elasticity and scalability from the outset. Not all systems need to implement all these to the same degree, specific project requirements will determine where effort is most wisely spent, but in the vast majority of cases the need to go reactive will demand that we design our applications differently.
In this presentation we explore several architecture elements that are commonly found in reactive systems (like the circuit breaker, various replication techniques, or flow control protocols). These patterns are language agnostic and also independent of the abundant choice of reactive programming frameworks and libraries, they are well-specified starting points for exploring the design space of a concrete problem: thinking is strictly required!
Reactive Streams: Handling Data-Flow the Reactive WayRoland Kuhn
Building on the success of Reactive Extensions—first in Rx.NET and now in RxJava—we are taking Observers and Observables to the next level: by adding the capability of handling back-pressure between asynchronous execution stages we enable the distribution of stream processing across a cluster of potentially thousands of nodes. The project defines the common interfaces for interoperable stream implementations on the JVM and is the result of a collaboration between Twitter, Netflix, Pivotal, RedHat and Typesafe. In this presentation I introduce the guiding principles behind its design and show examples using the actor-based implementation in Akka.
Reactive Programming, Traits and Principles. What is Reactive, where does it come from, and what is it good for? How does it differ from event driven programming? It only functional?
Distributed systems vs compositionalityRoland Kuhn
Distributed systems are becoming more and more commonplace, microservices and cloud deployments force this notion into the day to day routine of many developers. One of the great features of a strongly typed language like Scala—with its expressive type system—is that we can achieve a high level of safety and confidence by letting the compiler verify that our code behaves as specified. But can this safety be carried over into the interactions between distributed parts of an application? Can we for example safely compose Actor behaviours from different pieces?
This presentation introduces some approaches to this problem, including Session Types and π-calculus, and discusses their limitations. The practical ramifications are illustrated using Akka Typed, with a preview of composable and reusable behaviors.
Low latency microservices in java QCon New York 2016Peter Lawrey
In this talk we explore how Microservices and Trading System overlap and what they can learn from each other. In particular, how can we make microservices easy to test and performant. How can Trading System have shorter time to market and easier to maintain.
Multi-threading in the modern era: Vertx Akka and QuasarGal Marder
Everybody wants scalable systems. However, writing non-blocking applications in Java is not an easy task. In this session, we'll go over 3 different frameworks for managing multi-treading and concurrency support (Akka, Vertx and Quasar).
Reactive Java: Promises and Streams with Reakt (JavaOne talk 2016)Rick Hightower
see labs at https://github.com/advantageous/j1-talks-2016
Import based on PDF. This is from our JavaOne Talk 2016 on Reakt, reactive Java programming with promises, circuit breakers, and streams. Reakt is a reactive Java lib that provides promises, streams, and a reactor to handle asynchronous call coordination. It was influenced by the design of promises in ES6. You want to async-call serviceA and then serviceB, take the results of serviceA and serviceB, and then call serviceC. Then, based on the results of call C, call D or E and then return the results to the original caller. Calls to A, B, C, D, and E are all async calls, and none should take longer than 10 seconds. If they do, then return a timeout to the original caller. The whole async call sequence should time out in 20 seconds if it does not complete and should also check for circuit breakers and provide back pressure feedback so the system does not have cascading failures. Learn more in this session.
Time and ordering in streaming distributed systemsZhenzhong Xu
My GOTOcph 2018 talk
In the world of stream processing, distributed systems are composed from basic building blocks: coordination, transport, storage, and checkpointed compute services. Working together harmoniously, systems remain consistent, resilient and reliable. However, there is something far more fundamental at play: time and ordering; something experienced so tangibly, yet so difficult to solve for.
Over the past years, we at Netflix have learned to build an efficient, scalable and fault tolerant stream processing platform atop a microservices architecture. In this talk, we’ll try to examine the concepts of time, ordering, and causality in distributed systems. Wielding the knowledge of fundamental physics and computer science, we’ll look into how Einstein's Special Theory of Relativity, Lamport’s famous time clock paper, details of time synchronization, etc and their corresponding implications on streaming system design. Followed by evaluations of assumptions from some pre-streaming systems, then we’ll share the learnings from our hardest quests towards correct and consistent stream processing in the absence of ordering guarantees.
Spring Boot Microservices vs Akka Actor Cluster OpenCredo
Lorenzo Nicora introduces reactive principles and compares two different approaches to them: a microservice architecture based on Spring Boot and a clustered application using Akka, based on lessons learned from real-world projects. Lorenzo also briefly introduces the Actor programming model and how it differs from other approaches for tackling concurrent and non-blocking programming in Java.
How to implements the infra-structure for micro services? In this session, we'll compare building it with Spring Cloud vs. using Docker swarm/Kubernetes support.
How are systems in finance design for deterministic outcomes, and performance. What are the benefits and what is the performance you can achieve.
Included a demo you can download.
Microservices for performance - GOTO Chicago 2016Peter Lawrey
How do Microservices and Trading Systems overlap?
How can one area learn from the other?
How can we test components of microservices?
Is there a library which helps us implement and test these services?
This is a talk that I gave at the San Francisco DevOps meetup on 9/29/15. I talk about how Yelp performs service discovery using SmartStack and Docker.
Introduction to Akka 2. Explains what Akka's actors are all about and how to utilize them to write scalable and fault-tolerant systems.
Talk given at JavaZone 2012.
Lightbend Lagom: Microservices Just Rightmircodotta
Microservices architecture are becoming a de-facto industry standard, but are you satisfied with the current state of the art? We are not, as we believe that building microservices today is more challenging than it should be. Lagom is here to take on this challenge. First, Lagom is opinionated and it will take some of the hard decisions for you, guiding you to produce microservices that adheres to the Reactive tenents. Second, Lagom was built from the ground up around you, the developer, to push your productivity to the next level. If you are familiar with the Play Framework's development environment, imagine that but tuned for building microservices; we are sure you are going to love it! Third, Lagom comes with batteries included for deploying in production: going from development to production could not be easier. In this session, you will get an introduction to the Lightbend Lagom framework. There will be code and live demos to show you in practice how it works and what you can do with it, making you fully equipped to build your next microservices with Lightbend Lagom!
CQRS (Command Query Responsibility Segregation) was all the hype in .NET architecture circles a few years back. But has it faded away? Is it old news? I argue that it hasn't, and the concepts of CQRS are alive and well and probably more widely accepted and considered today than a few years ago. From event-driven systems to the Reactive Manifesto, the principles of CQRS are with us and impacting many different tools. In this session, we'll explore those CQRS principles and how they have manifested themselves in the architectures of choice today. You'll come away with a greater appreciation of CQRS and ideas on how to incorporate those principles in your applications today.
Modern HA applications in nowadays are developed with set of small focused and discrete Microservices. It's a trending concept and opens/solves questions like maintenance, scaling, live-deployments, security, fault-tolerance etc.
Distributed systems vs compositionalityRoland Kuhn
Distributed systems are becoming more and more commonplace, microservices and cloud deployments force this notion into the day to day routine of many developers. One of the great features of a strongly typed language like Scala—with its expressive type system—is that we can achieve a high level of safety and confidence by letting the compiler verify that our code behaves as specified. But can this safety be carried over into the interactions between distributed parts of an application? Can we for example safely compose Actor behaviours from different pieces?
This presentation introduces some approaches to this problem, including Session Types and π-calculus, and discusses their limitations. The practical ramifications are illustrated using Akka Typed, with a preview of composable and reusable behaviors.
Low latency microservices in java QCon New York 2016Peter Lawrey
In this talk we explore how Microservices and Trading System overlap and what they can learn from each other. In particular, how can we make microservices easy to test and performant. How can Trading System have shorter time to market and easier to maintain.
Multi-threading in the modern era: Vertx Akka and QuasarGal Marder
Everybody wants scalable systems. However, writing non-blocking applications in Java is not an easy task. In this session, we'll go over 3 different frameworks for managing multi-treading and concurrency support (Akka, Vertx and Quasar).
Reactive Java: Promises and Streams with Reakt (JavaOne talk 2016)Rick Hightower
see labs at https://github.com/advantageous/j1-talks-2016
Import based on PDF. This is from our JavaOne Talk 2016 on Reakt, reactive Java programming with promises, circuit breakers, and streams. Reakt is a reactive Java lib that provides promises, streams, and a reactor to handle asynchronous call coordination. It was influenced by the design of promises in ES6. You want to async-call serviceA and then serviceB, take the results of serviceA and serviceB, and then call serviceC. Then, based on the results of call C, call D or E and then return the results to the original caller. Calls to A, B, C, D, and E are all async calls, and none should take longer than 10 seconds. If they do, then return a timeout to the original caller. The whole async call sequence should time out in 20 seconds if it does not complete and should also check for circuit breakers and provide back pressure feedback so the system does not have cascading failures. Learn more in this session.
Time and ordering in streaming distributed systemsZhenzhong Xu
My GOTOcph 2018 talk
In the world of stream processing, distributed systems are composed from basic building blocks: coordination, transport, storage, and checkpointed compute services. Working together harmoniously, systems remain consistent, resilient and reliable. However, there is something far more fundamental at play: time and ordering; something experienced so tangibly, yet so difficult to solve for.
Over the past years, we at Netflix have learned to build an efficient, scalable and fault tolerant stream processing platform atop a microservices architecture. In this talk, we’ll try to examine the concepts of time, ordering, and causality in distributed systems. Wielding the knowledge of fundamental physics and computer science, we’ll look into how Einstein's Special Theory of Relativity, Lamport’s famous time clock paper, details of time synchronization, etc and their corresponding implications on streaming system design. Followed by evaluations of assumptions from some pre-streaming systems, then we’ll share the learnings from our hardest quests towards correct and consistent stream processing in the absence of ordering guarantees.
Spring Boot Microservices vs Akka Actor Cluster OpenCredo
Lorenzo Nicora introduces reactive principles and compares two different approaches to them: a microservice architecture based on Spring Boot and a clustered application using Akka, based on lessons learned from real-world projects. Lorenzo also briefly introduces the Actor programming model and how it differs from other approaches for tackling concurrent and non-blocking programming in Java.
How to implements the infra-structure for micro services? In this session, we'll compare building it with Spring Cloud vs. using Docker swarm/Kubernetes support.
How are systems in finance design for deterministic outcomes, and performance. What are the benefits and what is the performance you can achieve.
Included a demo you can download.
Microservices for performance - GOTO Chicago 2016Peter Lawrey
How do Microservices and Trading Systems overlap?
How can one area learn from the other?
How can we test components of microservices?
Is there a library which helps us implement and test these services?
This is a talk that I gave at the San Francisco DevOps meetup on 9/29/15. I talk about how Yelp performs service discovery using SmartStack and Docker.
Introduction to Akka 2. Explains what Akka's actors are all about and how to utilize them to write scalable and fault-tolerant systems.
Talk given at JavaZone 2012.
Lightbend Lagom: Microservices Just Rightmircodotta
Microservices architecture are becoming a de-facto industry standard, but are you satisfied with the current state of the art? We are not, as we believe that building microservices today is more challenging than it should be. Lagom is here to take on this challenge. First, Lagom is opinionated and it will take some of the hard decisions for you, guiding you to produce microservices that adheres to the Reactive tenents. Second, Lagom was built from the ground up around you, the developer, to push your productivity to the next level. If you are familiar with the Play Framework's development environment, imagine that but tuned for building microservices; we are sure you are going to love it! Third, Lagom comes with batteries included for deploying in production: going from development to production could not be easier. In this session, you will get an introduction to the Lightbend Lagom framework. There will be code and live demos to show you in practice how it works and what you can do with it, making you fully equipped to build your next microservices with Lightbend Lagom!
CQRS (Command Query Responsibility Segregation) was all the hype in .NET architecture circles a few years back. But has it faded away? Is it old news? I argue that it hasn't, and the concepts of CQRS are alive and well and probably more widely accepted and considered today than a few years ago. From event-driven systems to the Reactive Manifesto, the principles of CQRS are with us and impacting many different tools. In this session, we'll explore those CQRS principles and how they have manifested themselves in the architectures of choice today. You'll come away with a greater appreciation of CQRS and ideas on how to incorporate those principles in your applications today.
Modern HA applications in nowadays are developed with set of small focused and discrete Microservices. It's a trending concept and opens/solves questions like maintenance, scaling, live-deployments, security, fault-tolerance etc.
While we’re drawing ever closer to Java 9, and even hearing about features in Java 10, it’s also true that many of us are still working with an older version. Even if your project has technically adopted Java 8, and even if you’re using it when coding new features, it’s likely the majority of your code base is still not making the most of what’s available in Java 8 - features like Lambda Expressions, the Streams API, and new Date/Time.
In this presentation, Trisha:
- Highlights some of the benefits of using Java 8 - after all, you’ll probably have to persuade The Management that tampering with existing code is worthwhile
- Demonstrates how to identify areas of code that can be updated to use Java 8 features
- Shows how to automatically refactor your code to make use of features like lambdas and streams.
- Covers some of the pros and cons of using the new features - including suggestions of when refactoring may NOT be the best idea.
Introduction to Amazon Web Services - How to Scale your Next Idea on AWS : A ...Amazon Web Services
Building powerful web applications in the AWS Cloud : A Love Story, Design patterns in web-based cloud architecture, Jinesh Varia gave this talk at Cloud Connect and several other places
http://aws.typepad.com/aws/2011/03/building-powerful-web-applications-in-the-aws-cloud-a-love-story.html
AWS re:Invent 2016: Running Microservices on Amazon ECS (CON309)Amazon Web Services
Running and managing large scale applications with microservices architectures is hard and often requires operating complex container management infrastructure. Amazon EC2 Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances. In this session, we will walk through a number of patterns used by our customers to run their microservices platforms. We will dive deep into some of the challenges of running microservices, such as load balancing, service discovery, and secrets management, and we’ll see how Amazon ECS can help address them. We'll also hear from Instacart how they use a blue/green deployment process to deploy services to ECS and how they manage configuration with a RDS-based metadata service.
Indeed Flex: The Story of a Revolutionary Recruitment PlatformHostedbyConfluent
"This is a tale of two streams when the pandemic hit and how we changed with the times and built a revolutionary recruitment platform for “going into work”. We engaged employers, recruiters and job seekers from industrial, healthcare, retail, hospital and facilities management sectors by building a unique platform where the job seeker has full control to pick their schedule, pay rate and what meets their preferences. Our goals are to give job seekers and employers a platform that thrives on simplicity, transparency and low costs. The Flexer stands today with full control of their time at the edge of opportunities to thrive on.
This presentation will go into the details of how we are tearing down a monolithic platform piece by piece and building a robust architecture,
- Routing events between two platforms
- Many sources and,
- Consumed by downstream several applications
We will discuss the caveats and bugs we learned when we worked with schema registry and evolution of schemas. We will highlight improvements we gained from automation and observability with Datadog integration for Confluent Cloud.
If you’re in discussions surrounding event driven systems at your organization then this talk is for you. Join Ronak and me for this talk and let’s have a discussion."
HPE’s Erik Vogel on Key Factors for Driving Success in Hybrid Cloud Adoption ...Dana Gardner
A discussion on innovation around maturing hybrid cloud models and how proper common management of hybrid cloud operations makes or breaks the expected benefits.
Strata SC 2014: Apache Mesos as an SDK for Building Distributed FrameworksPaco Nathan
O'Reilly Media - Strata SC 2014
Apache Mesos is an open source cluster manager that provides efficient resource isolation for distributed frameworks—similar to Google’s “Borg” and “Omega” projects for warehouse scale computing. It is based on isolation features in the modern kernel: “cgroups” in Linux, “zones” in Solaris.
Google’s “Omega” research paper shows that while 80% of the jobs on a given cluster may be batch (e.g., MapReduce), 55-60% of cluster resources go toward services. The batch jobs on a cluster are the easy part—services are much more complex to schedule efficiently. However by mixing workloads, the overall problem of scheduling resources can be greatly improved.
Given the use of Mesos as the kernel for a “data center OS”, two additional open source components Chronos (like Unix “cron”) and Marathon (like Unix “init.d”) serve as the building blocks for creating distributed, fault-tolerant, highly-available apps at scale.
This talk will examine case studies of Mesos uses in production at scale: ranging from Twitter (100% on prem) to Airbnb (100% cloud), plus MediaCrossing, Categorize, HubSpot, etc. How have these organizations leveraged Mesos to build better, more scalable and efficient distributed apps? Lessons from the Mesos developer community show that one can port an existing framework with a wrapper in approximately 100 line of code. Moreover, an important lesson from Spark is that based on “data center OS” building blocks one can rewrite a distributed system much like Hadoop to be 100x faster within a relatively small amount of source code.
These case studies illustrate the obvious benefits over prior approaches based on virtualization: scalability, elasticity, fault-tolerance, high availability, improved utilization rates, etc. Less obvious outcomes also include: reduced time for engineers to ramp-up new services at scale; reduced latency between batch and services, enabling new high-ROI use cases; and enabling dev/test apps to run on a production cluster without disrupting operations.
Ever wonder how some applications are built? Ever wonder how to combine components of the Windows Azure platform? Stop wondering and learn how we’ve built MyGet.org, a multi-tenant software-as-a-service. In this session we’ll discuss architecture, commands, events, access control, multi tenancy and how to mix and match those things together. Learn about the growing pains and misconceptions we had on the Windows Azure platform. The result just may be a reliable, cost-effective solution that scales.
Getting Started with Docker - Nick StinematesAtlassian
Docker is an open-source engine that automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere. In this session, you will learn how to get started building your first Docker container, and how to use Docker containers to simplify your CI process.
Problems you’ll face in the Microservices World: Configuration, Authenticatio...Quentin Adam
Okay, Microservices are cool. But, as all the new trendy buzzword, it’s not a silver bullet, and there are several problems to manage. One is the authentication, distributed authentication is hard, and there is many ways to achieve it. Configuration is the second issue to be managed when dealing with distributed micro application strategy. This talk is a concrete return of experience to build a strategy on microservice and problems we will have to deal on this occasion.
Presentation from DDD Sydney, May 28th, 2016
Buzz word! More buzz words! And another buzz word!! Now that that's out of the way, if you're thinking of heading down the microservices path, then how do you do it? How do you build the services? What do you need to think about if you're starting from scratch? What if you're converting a legacy app? How do we deal with versioning? Do we have to use a NoSQL solution, just because Netflix does? Do we need to use docker/containers? What about the code? Show me the code! Well, that's what this session is all about. Designing and building microservices in .NET and then handling a bunch of other concerns that a microservices approach will force you to think about. Sounds interesting, doesn't it? You betcha.
AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...GeeksLab Odessa
4.6.16 AI&BigData Lab
Upcoming events: goo.gl/I2gJ4H
Как устроить анализ данных 40 млн. человек за 5 лет так, чтобы это выглядело почти в реальном времени.
Spring I_O 2024 - Flexible Spring with Event Sourcing.pptxSteve Pember
Event Sourcing is a modern but non-trivial data model for building scalable and powerful systems. Instead of mapping a single Entity to a single row in a datastore, in an Event Sourced system we persist all changes for an Entity in an append-only journal. This design provides a wealth of benefits: a built-in Audit Trail, Time-Based reporting, powerful Error Recovery, and more. It creates flexible, scalable systems and can easily evolve to meet changing organizational demands. That is, once you have some experience with it. Event Sourcing is straightforward in concept, but it does bring additional complexity and a learning curve that can be intimidating. People coming from traditional ORM systems often wonder: how does one model relations between Entities? How is Optimistic Locking handled? What about datastore constraints?
Based on over eight years of experience with building ES systems in Spring applications, we will demonstrate the basics of Event Sourcing and some of the common patterns. We will see how simple it can be to model events with available tools like Spring Data JPA, JOOQ, and the integration between Spring and Axon. We’ll walk through sample code in an application that demonstrates many of these techniques. However, it’s also not strictly about the code; we’ll see how a process called Event Modeling can be a powerful design tool to align Subject Matter Experts, Product, and Engineering. Attendees will leave with an understanding of the basic Event Sourcing patterns, and hopefully a desire to start creating their own Journals.
Anatomy of a Spring Boot App with Clean Architecture - Spring I/O 2023Steve Pember
In this presentation we will present the general philosophy of Clean Architecture, Hexagonal Architecture, and Ports & Adapters: discussing why these approaches are useful and general guidelines for introducing them to your code. Chiefly, we will show how to implement these patterns within your Spring (Boot) Applications. Through a publicly available reference app, we will demonstrate what these concepts can look like within Spring and walkthrough a handful of scenarios: isolating core business logic, ease of testing, and adding a new feature or two.
SACon 2019 - Surviving in a Microservices EnvironmentSteve Pember
Many presentations on microservices offer a high-level view of the architecture; rarely do you hear what it’s like to work in such an environment. Stephen Pember shares his experience migrating from a monolith to microservices across several companies, highlighting the mistakes made along the way and offering advice.
Surviving in a Microservices environment -abridgedSteve Pember
Many presentations on Microservices offer a high-level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some considerable DevOps work and start learning about deployment pipelines, metrics, and logging.
Don’t panic. In this presentation we’ll discuss what we learned over the past four years by highlighting our mistakes. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll see how team communication is more important than one might realize. Most importantly, we’ll show how - while an individual service is simple - the infrastructure demands are now much more complicated: your organization will need to introduce and become increasingly dependent on various technologies, procedures, and tools - ranging from the ELK stack to Grafana to Kubernetes. Lastly, you’ll come away with the understanding that your resident SREs will become the most valued members of your team.
Over the past few years, Gradle has become a popular build tool in the JVM space. This is not surprising, considering the power and the features it brings, compared with its competitors. However, one thing Gradle lacks is history and the collective knowledge at the same level of other alternatives: how does one organize a Gradle project in an ‘idiomatic’ fashion?
We feel that we’ve put together a decent build pipeline for each of our microservices over the years, and each one starts with their build.gradle file(s). We’d like to share it, although we’re not sure if it’s the ‘correct’ way.
In this talk, we’ll walk through a sample project structure and build process. We’ll discuss the various checks and tools we use (e.g. Sonar, CodeNarc, Jenkins) at each step of the build. We’ll explain how each of the components in the process work for us, and share samples of our Groovy scripts. Most importantly, though, we’d like to hear what the audience are using in their builds!
Many presentations on Microservices offer a high level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some DevOps work and start learning about deployment pipelines, metrics, and logging.
Don’t panic. In this presentation we’ll discuss what we, at ThirdChannel, learned over the past four years. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll dive a bit into DevOps and see how one will become dependent on various metric and centralized logging tools, like Kubernetes and the ELK stack. Finally we’ll talk about team communication and organization… and how they are likely the most important tool for surviving a Microservices development team.
Reactive applications & reactive programming result in flexible, concise, performant code and are a superior alternative to the old thread-based programming model. The reactive approach has gained popularity for a simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in reactive terms, particularly when one realizes that we must be Reactive up and down the entire programming stack.
In this talk we’ll explore what it means to be ‘Reactive’. We’ll examine some of the more interesting tools available to us, some of which come from the Groovy community. Specifically we’ll cover Ratpack, RxGroovy, React, and RabbitMq - along with examples and a sample implementation. We’ll demonstrate how effectively they can work together at each level of the stack - from the front end, to the back end, to handling http requests and message queue events - and how easy it can be to go Reactive all the way down.
Event storage offers many practical benefits to distributed systems providing complete state changes over time, but there are a number of challenges when building an event store mechanism. Stephen Pember explores some of the problems you may encounter and shares real-world patterns for working with event storage.
Harnessing Spark and Cassandra with GroovySteve Pember
This talk is an introduction to a powerful combination in the big data space: Apache Spark and Cassandra. Spark is a cluster-computing framework that allows users to perform calculations against resilient in-memory datasets using a functional programming interface. Cassandra is a linearly scalable, fault tolerant, decentralized datastore. These two technologies are complicated, but integrate well and provide such a level of utility that whole companies have formed around them.
In this talk we’ll learn how Spark and Cassandra can be leveraged within your Groovy Application: Spark normally asks for a Scala environment. We’ll talk about Spark and Cassandra from a high level and walk through code examples. We’ll discuss the pitfalls of working with these technologies - like modeling your data appropriately to ensure even distribution in Cassandra and general packaging woes with Spark - and ways to avoid them. Finally, we’ll explore how we at ThirdChannel are using these technologies.
Surviving in a microservices environmentSteve Pember
Many presentations on Microservices offer a high level view; rarely does one hear what it’s like to work in such an environment. Individual services are somewhat trivial to develop, but now you suddenly have countless others to track. You’ll become obsessed over how they communicate. You’ll have to start referring to the whole thing as “the Platform”. You will have to take on some DevOps work and start learning about deployment pipelines, metrics, and logging.
Don’t panic. In this presentation we’ll discuss what we learned over the past three years. We’ll examine what a development lifecycle might look like for adding a new service, developing a feature, or fixing bugs. We’ll dive a bit into DevOps and see how one will become dependent on various metric and centralized logging tools, like Kubernetes and the ELK stack. Finally we’ll talk about team communication and organization... and how they are likely the most important tool for surviving a Microservices development team.
Surviving in a Microservices EnvironmentSteve Pember
Cloud Native Microservice architectures have become increasingly popular over the past few years, and for good reasons: smaller, efficient codebases, finely targeted scaling options, and the ability to do continuous deployment along with continuous integration, among others. All potentially very powerful features. However - as with most things - Microservices bring tradeoffs in terms of application complexity: working with an individual service is easy; overall application development becomes increasingly complex. Perhaps too complex for your average web application.
Many presentations on the Microservice phenomena offer either a high level view on what it is, compare and contrast it with the Monolith pattern, or discuss how to migrate from a Monolith to Microservices, but rarely does one hear what it’s like to actually work in such an environment. Frankly, it can be intimidating for someone accustomed to a traditional monolithic development experience. Individual services are somewhat trivial to develop, but now you suddenly have countless others to keep track of. You may become lost: with all these services, is anyone directing the overall development? You’ll become obsessed over how and when they communicate. You’ll have to start referring to the application on the whole as “the Platform”. It’ll soon become difficult or even impossible to run the whole Platform on a development laptop. You may even have to take on some DevOps work, and start learning about deployment pipelines, and whole new worlds of metrics and logging.
Don’t panic. In this presentation we’ll discuss what we learned working with a Microservice platform for the past three years. We’ll cover what to expect when joining a Microservice team and what the situation will look like as the team size grows. We’ll see how critical inter-service testing strategies are to the success of the team. We’ll examine what a development lifecycle might look like for adding a new service, developing a new feature, or fixing bugs. We’ll dive a bit into DevOps and see how one will become dependent and various metric and centralized logging tools, like Kubernetes and the ELK stack. Finally we’ll talk about communication, team organization strategies, and how they are likely the most important tool for surviving a Microservices development team.
An introduction to Reactive applications, Reactive Streams, and options for t...Steve Pember
The term “reactive” has lately become a buzzword, with a variety of definitions around the Web. When you hear “reactive,” what do you think of? Reactive Streams? The Reactive Manifesto? ReactJS? These terms may seem unrelated, but they share a common core concept.
Reactive applications and Reactive programming result in flexible, concise, performant code and are a superior alternative to the old, standard thread-based imperative programming model. The Reactive approach has gained popularity recently for one simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in Reactive terms due to how accustomed we’ve become to the imperative style.
Stephen Pember explores the various definitions of Reactive and Reactive programming with the goal of providing techniques for building efficient, scalable applications. Steve dives into the key concepts of Reactive Streams and examines some sample implementations—including how ThirdChannel is currently using reactive libraries in production code. Steve looks at some of the open source options available in the JVM—including Reactor, RxJava, and Ratpack—giving you an idea of where to begin with the reactive ecosystem. If Reactive is new to you, this should be an excellent introduction.
Are you tired of Hibernate? Is GORM is too heavy for your current project? Do you like having more control over your SQL? Do you like flexible DSLs? Try JOOQ!
JOOQ (Java Object Oriented Querying) is light-weight alternative to classic data access solutions or ORMs like Hibernate, JPA, JDBC, and GORM. JOOQ's goal is to give the developer a flexible DSL for building typesafe, database agnostic SQL queries, and attempts to convince the developer of a ‘database-first’ approach to building their application. In this talk we’ll quickly present an introduction to JOOQ from a high level, discuss its features, and see several examples of how we’re using JOOQ to great effect with many Spring Boot and Ratpack apps within our platform.
Reactive Streams and the Wide World of GroovySteve Pember
The concept of Reactive Streams (aka Reactive Extensions, Reactive Functional Programming, or simply Rx) has become increasingly popular recently, and with good reason. The Reactive Streams specification provides a universal abstraction for asynchronously processing data received across multiple sources (e.g. database, user input, third-party services), and includes mechanisms for controlling the rate at which data is received. This makes it a powerful tool within a Microservice platform. And did we mention that the Groovy lang community is quite involved?
In this talk we’ll explore the various features and concepts of Reactive Streams. We’ll talk about some typical use cases for Rx and more importantly, how to implement them. We’ll focus primarily on RxGroovy and Ratpack, then provide example implementations that show you how to get started with this powerful technique.
As businesses grow, so does the complexity of their software. New features, new models, and new background processes all continue to be added. . .and developers struggle to make sense of it all. Yet the end user demands a swift and functional experience when interacting with your application. It is paramount to be open to alternative patterns that help tame complex, high-demand services. Two such patterns are command-query responsibility segregation (CQRS) and event sourcing (ES).
Command-query responsibility segregation is an architectural pattern for user-facing applications that extends from the now standard Model-View-Controller (MVC) pattern and is an alternative to the CRUD pattern. At its core, CQRS is about changing how we think of and work with our data by introducing two types of models: all user actions become commands, and a read-only query model powers our views. Commands and queries are logistically separated, providing additional decoupling of our application. CQRS also calls for changes in how we store and structure our data.
Enter event sourcing. Instead of persisting the current state of our domain objects or entities, we record historical events about our data. The key advantage is that we can examine our application data at any point in time, rather than just the current state. This pattern changes how we persist and process our data but is surprisingly efficient.
While each of the two patterns can be used exclusively, they complement each other beautifully and facilitate the construction of decoupled, scalable applications or individual services. Stephen Pember explores the fundamentals of each pattern and offers several examples and demonstration code to show how one might actually go about implementing CQRS and ES. Steve discusses task-based UIs and domain-driven design as he outlines some of the advantages—and challenges—that ThirdChannel has seen when developing systems using CQRS and ES over the past year.
An Introduction to Reactive Application, Reactive Streams, and options for JVMSteve Pember
The term “reactive” has lately become a buzzword, with a variety of definitions around the Web. When you hear reactive, what do you think of? Reactive Streams? The Reactive Manifesto? ReactJS? These terms may seem unrelated, but they share a common core concept.
Reactive applications and reactive programming result in flexible, concise, performant code and are a superior alternative to the old standard thread-based imperative programming model. The reactive approach has gained popularity recently for one simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in reactive terms due to how accustomed we’ve become to the imperative style.
Stephen Pember explores the various definitions of reactive and reactive programming with the goal of providing techniques for building efficient, scalable applications. Steve dives into the key concepts of Reactive Streams and examines some sample implementations—including how ThirdChannel is currently using reactive libraries in production code. Steve looks at some of the open source options available in the JVM—including Reactor, RxJava, and Ratpack—giving attendees an idea of where to begin with the reactive ecosystem. If reactive is new to you, this should be an excellent introduction.
Richer Data History with Event Sourcing (SpringOne 2GX 2015Steve Pember
A common pattern in application development is to build systems where the data is directly linked to the current state of the application; one row in the database equates to one entity’s current state. Only ever knowing the current state of the data is adequate for many systems, but imagine the possibilities if one had access to the state of the data at any point in time. Enter Event Sourcing: instead of persisting the current state of our Domain Objects or Entities, we record historical events about our data. This pattern changes how we persist and process our data, but is surprisingly lightweight. In this talk I will present the basic concepts of Event Sourcing and the positive effects it can have on analytics and performance. We’ll discuss how storing historical events provides extremely powerful views into our data at any point in time. We’ll see how naturally it couples with the Event-oriented world of modern Reactive systems, and how easily it can be implemented in Groovy. We’ll examine some practical use cases and example implementations in Ratpack. Event Sourcing will change how you think about your data.
Springone2gx 2015 Reactive Options for GroovySteve Pember
Reactive applications and Reactive programming are an alternative to the standard thread-based imperative programming model that can result in flexible, concise code. The Reactive approach has gained popularity recently for one simple reason: we need alternative designs and architectures to meet today’s demands. However, it can be difficult to shift one’s mind to think in Reactive terms. It doesn’t help that the descriptions around the web can be contradictory and the library documentation can be obscure. In this talk, we’ll explore the concepts of Reactive and Reactive Programming. We’ll demonstrate some of the useful Reactive functions and examine some practical implementations - including how we’re currently using Reactive libraries in production code. Most importantly, we’ll look at some of the open source options available to us in the Groovy community, including Reactor, RxJava, and Ratpack. If Reactive is new to you, this should be an excellent introduction.
Gr8conf US 2015 - Intro to Event Sourcing with GroovySteve Pember
As Grails developers we generally build systems where our data is directly linked to the current state of our application; one row in the database equates to one entity’s current state. Only ever knowing the current state of the data is adequate for many systems, but imagine the possibilities if one had access to the state of the data at any point in time.
Enter Event Sourcing: instead of persisting the current state of our Domain Objects, we store historical events about our data. This pattern changes how we store and process our data, but is surprisingly lightweight and performant. In this talk I will present the basic concepts of Event Sourcing and the positive effects it can have on analytics and performance. We’ll see how naturally it couples with the Event-oriented world of modern Reactive systems, and how easily it can be implemented in Groovy. We’ll examine some practical use cases and example implementations. Event Sourcing will change how you think about your data.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
93. Image Credits
Complexity (pipes): http://www.flickr.com/photos/bhophoto/384574407/
Simple Pipeline: http://tierracos.com/our-companies/tierra-pipeline/
Highway: http://farnorthdallas.advocatemag.com/wp-content/uploads/2013/04/882349_500402383328407_539732406_o.jpg
Homer & doughnuts: http://thechurchillreview.blogspot.com/2012/10/feeling-terror-too-young-aka-kiddie.html
Tom Brady: http://stamperdad.wordpress.com/category/tom-brady/
Telephone Exchange Operator: http://fineartamerica.com/featured/telephone-exchange-1920s-granger.html
People in Queue: http://www.guardian.co.uk/money/2010/mar/23/dole-queue-jobseekers-online
Cookie Monster: http://muppet.wikia.com/wiki/Cookie_Monster_Through_the_Years
Scrooge McDuck: http://smallbusiness.uprinting.com/how-pennies-are-hurting-small-business/
Too Many Cooks: http://www.ecommercesystems.com/featured-articles/cooks-kitchen-driving-ecommerce-business/
Clustered Macs: http://www.uiowa.edu/mihpclab/micg.html
Resuscitate:
http://www.dailymail.co.uk/health/article-2034160/Do-resuscitate-Theyre-fateful-words-meaning-doctors-wont-try-save-you-collapse-hospital.h
Iron Man: http://images.wikia.com/marvelmovies
Mail Sorting: http://riversidechamber.files.wordpress.com
Twitter Logo & app chart: https://blog.twitter.com/engineering
Modern Server Farm: http://bookriot.com/2013/03/26/book-less-libraries-and-other-contemporary-realities/
Grey World: http://upload.wikimedia.org/wikipedia/commons/d/d0/BlankMap-World-1ce.png
Editor's Notes
-I won’t try to make anyone guess what this is
-I should explain it a bit, as I’ll be following a similar format for other diagrams.
-Blue symbol represents some feature - browsing, managing, payments, cart
-black box is a single web node or application
-cloud is internet
Let’s talk about complexity and scalability
-A few questions, before we begin.
-I’d like for you to keep the answers to these in your head as we go along
-or rather, is everything in trunk / master?
-Have you ever had to find the needle in that giant haystack?
-do you miss TDD?
whether it be a total overhaul or just a few features.
And if so, did it take you longer than coding that new feature?
These are all symptoms of a Monolithic application, If you answered “YES”, then you probably have a Monolithic application
“convienent”
The Complexity will become enormous. Any gains you think you may have made at the outset will not continue, particularly once you start to attract users.
..And grow your team.
- A monothlic app will not scale!
You may say to yourself, “Wait, Steve!” what do you mean it won’t scale?
I can load balance my nodes...
Add Swimlanes, master-slave dbs in case that doesn’t work
-or add caching in between my nodes
Can you really look at a class or schema diagram like this and really think, “Yeah, that’s awesome. Nailed it.”?
-this was the largest I could find on Google images. I’m sure we’ve all seen the massive versions of this that cover multiple pages, taped to the wall.
- If you ever have to tape to the wall, done something wrong
The ability to add new features, fix bugs, or resolve technical debt cannot scale with the size of the application. Throwing more developers at it also will not help
- One of the best passive-aggressive statements I’ve seen
- Passive aggressively scrawled on the wall: You can’t make a baby in 1 month with 9 women.
-And trying to refactor anything?
As the size of your codebase increases, the computational complexity will exponentially increase and will have adverse effects on maintenance. This has been known for some time; there’s been papers written on the subject. Although it’s difficult to measure this.
This is such a misguided architecture, it makes you wonder how exactly we collectively ended up here.
I blame it largely at the feet of the large, popular web frameworks, e.g. Rails, Django, Grails.
Don’t get me wrong, they have their uses
But they’re touted as if they’re the magic cure for building your app or product; that your company should be based entirely within a framework. You hear folks say, “Oh, what are you using in your startup?” “We’re a Rails shop, obviously!”.
I argue that your framework choice is largely irrelevant. When people ask “what technology are you using?” the answer should be something like “Ugh, well, that’s difficult”.
There have been attempts to mitigate
-Let’s take our anecdotal design, an e-commerce store.
With SOA, one would break up each individual component, and setup communication between each service over some direct service calls. Eg SOAP or REST.
-Note that these calls are traditionally Synchronous over http, so it’s imperative that each node return as quickly as possible (that whole ‘responsive’ notion from the Manifesto)
Having disparate services keeps the individual components easier to manage and keep track of, which reduces overall complexity dramatically
… as each feature set will be in its own simpler repository
With distributed applications, each node becomes its own concentrated ecosystem. New features can be added with relatively little pain.
And the code is easier to maintain. I highlight this point for a reason. Bugs are vastly easier to discover. Refactoring is highly concentrated and (shouldn’t) disrupt the work of the other nodes.
and other numbers that managers love
-organizing the event based communication and see things flow and respond, it’s magical
possibly delete
However! (There’s always a downside, eh?)
You’ll need someone to oversee the team and catch those engineers which are straying from the distributed vision
-find out how each will communicate
- converting to a distributed, services oriented app is an investment: a decent amount of upfront cost, but the dividends are fantastic.
HTTP calls are relatively slow
* Having a web of interconnected services can also grow out of control. Having to configure service nodes so that each knows where the others are located in the network can be cumbersome to manage.
In other words, SOA (along with the other architecture patterns that have attempted to fix monolithic) do a decent job...
... we can do better
So what does this have to do with the Reactive pattern?
being event driven increases decoupling of components
-Couldn’t think of proper images to describe these
-Adhering to these 4 tenants is all well and good
Furthermore: being Reactive is a development ethos and there’s not one right answer
So, going back to the original e-commerce example…
- how to go reactive?
First of course, is to break up our application into individual components, but this is not good enough.
Could still adhere to be resilient and responsive, and it’s certainly more scalable than it was.
Simply breaking them apart won’t make each service node resilient or responsive, that’s up to internal dev architecture (more from Dave)
In other words, extend the idea of having an event driven node or service into the notion of having event-driven communication between services.
-And how do we facilitate event based communication between our services?
One method: Message Brokers!
Your basic approach. A single Producer, a Queue of Messages, and a Consumer. In this case, quite the prodigious one.
Note the queue backup. He should probably share.
The key to a message oriented approach is when certain events occur to encapsulate data into individual messages and drop them on some queue or exchange
TODO: Image of Postman / message.
Can spin up or down nodes for targeted scaling of your services as demand dictates
-Example from before, an e-commerce site
-we’ve broken the features into individual applications and spun up one node per each
-connected the users’ cart to a queue, and our order processing into another queue
-suppose the cart and order processing start experiencing heavy load
- programatically spin up new instances to handle it
- Going back into the refactoring discussion -> Can pit different implementations of a service against each other via AB testing
TIME: Asynchronous
LOCATION: Nodes do not care where the other nodes are, only worry about the broker
CARDINALITY: Each service instance does not care about the number of others out there
There are quite a few to choose from. I point out 0mq, activemq, open amq, and my favorite…
Why?
uses the raw AMQP protocol (as do some of the others I mentioned)
the actual broker itself is a lightweight Erlang application
Offers two key security features: message persistance and error recovery
And it’s fast.
Although not nearly as fast as some other Brokers, specifically zeroMQ
If you’re struggling to process 4k+ messages/ second, that sounds like an awesome engineering problem and you contact us.
The previous description of message queues glosses over what is happening a bit.
-Messages are routed through an EXCHANGE in to one or many queues.
-Messages are then plucked off the queue by an attached, waiting Consumer. Or, the first available if multiple are attached.
-Exchanges can have different delivery techniques.
If a message is mishandled or the consumer crashes / has an error during message processing, message is returned to the queue for another consumer to pick it up or try again
Furthermore, Queues and exchanges can be made “durable”, meaning they’ll survive a restart or server crash with messages intact
RabbitMQ nodes can be clustered
-federated-> shared / copied
-shoveled -> pushed
There are several message delivery strategies one can set up for their exchanges
With the whole purpose of scaring you, here’s the Topic Queue, the most complicated
-gets complicated
-allows for all sorts of wild routing schemes
-all messages go to the BACK of the queue
-each queue could equate to namespaced events, like “order.fail” or “order.success”. A service could emit an ‘order.fail’ event for subscriber queues to receive
- A simpler concept is the direct exchange, where queues register on an exact String key, and messages state which key they should be ‘addressed’ to
I feel these are the easiest to use and the best way to get started, but consider the topic exchanges for true power
Here, too, a practical usage could be equate a queue key registration as an event name.
This is a depiction of a Work Queue, the most common configuration.
-An exchange routes messages via some method (topic, direct) to a queue with multiple consumers
-Work queues operate on a first-come-first serve approach
-I think this is the coolest, one can monitor the state of the exchange and programmatically spin up or down the number of consumer nodes
-All of these consumers are peers...
-Finally, there’s ‘fanout’, which distributes a single message to all attached queues. Good for a general broadcast notification / pub-sub to all your nodes.
-there needs to be a one-to-one queue to consumer relation, though, I believe.
-A special use, case, really.
- should say type=fanout under that X
…Twitter
They had one of the largest Rails deployments in the world.
Epitome of Monolithic.
Engineers needed to pull in Global Experts to get anything done on the codebase
Spent more time on “Archeological Digs” or “Whale Hunting expeditions” to track down bugs or learn how things worked than on developing new features
Turning point came during 2010 World Cup. Flurry of tweets brought twitter to its knees.
Service was unavailable, engineers worked through the night
Debated putting a vuvuzella sound effect here.
They shattered their monolithic application into multiple individual applications..
The size of the circles represent the number of deployments of that particular application
-I know I just spoke about Rabbit, but Twitter developed their own tool called Finagle which handles asynchronous communication between nodes
Each with their own small, determined, dedicated teams.
they switched from the Ruby Virtual Machine to the JVM, were no strangers to the JVM
-Search written in Java
Embraced Event-based programming from the bottom up
Went with Netty (as opposed to something like NodeJS), which is an NIO technology for the JVM
Netty is NIO
When Twitter was Monolithic, they were able to achieve about 200 to 300 requests or tweets per second per application node.
After going reactive, they’re up to 10 to 20 thousand. 2 orders or magnitude?
So, as of August 2013, Twitter handles an average of 5700 Tweets per minutes, and under load, can dynamically ramp that number up to 144k without any direct action by staff
-> Incidentally, JVM advocates and Rails detractors love to talk about how Twitter moved away from Rails to Java. And while that’s true that Java would give better performance, the Architecture choice was vastly more important
We didn’t want to go down that same old route.
-Second thing we did was take a step back and look at their current setup.