Downloaded 11 times
Uploaded byJudy Breedlove
App Mod 05: App resiliency detecting and preventing issues in distributed apps
The document discusses distributed services architectures and resilient distributed applications. It covers benefits of distributed architectures like performance, reliability and extensibility. It also discusses challenges of distributed systems like unpredictable failures, system degradation and elastic resources. The document then describes how tools like Istio can help with issues like circuit breaking, load balancing and failure recovery in distributed microservices applications. It outlines a lab experiment using Istio on OpenShift to visualize metrics, alter routing and inject faults in distributed applications.
More Related Content
![[Konveyor] migrate and modernize your application portfolio to kubernetes wit...](https://cdn.slidesharecdn.com/ss_thumbnails/konveyormigrateandmodernizeyourapplicationportfoliotokuberneteswithtackle-210728162324-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to App Mod 05: App resiliency detecting and preventing issues in distributed apps
App Mod 05: App resiliency detecting and preventing issues in distributed apps
- 1. MONOLITHS TO MICROSERVICES: APPTRANSFORMATION Hands-on Technical Workshop
- 2. PART 5: RESILIENTDISTRIBUTED APPS
- 3. INSERT DESIGNATOR, IFNEEDED3 DISTRIBUTED SERVICES ARCHITECTURES Benefits (when implemented correctly): ● Performance ● Reliability ● Resiliency ● Extensibility ● Availability ● Robustness
- 4. INSERT DESIGNATOR, IFNEEDED4 DISTRIBUTED SERVICES ARCHITECTURES Fallacies of Distributed Computing ● The network is reliable. ● Latency is zero. ● Bandwidth is infinite. ● The network is secure. ● Topology doesn't change. ● There is one administrator. ● Transport cost is zero. ● The network is homogeneous. wikipedia.org/wiki/Fallacies_of_distributed_computing
- 5. INSERT DESIGNATOR, IFNEEDED5 DISTRIBUTED SERVICES ARCHITECTURES Applications must deal with ● Unpredictable failure modes ● End-to-end application correctness ● System degradation ● Topology changes ● Elastic/ephemeral/transient resources A E B C F G D H I Client
- 6. INSERT DESIGNATOR, IFNEEDED6 POSSIBLE SOLUTIONS Today, Developers do this: ● Circuit Breaking ● Bulkheading ● Timeouts/Retries ● Service Discovery ● Client-side Load Balancing
- 7. INSERT DESIGNATOR, IFNEEDED7 TOO MUCH INFRASTRUCTURE IN BUSINESS LOGIC Platform Microservice Netflix OSS Netflix OSS Business Logic Container Platform Microservice Business Logic
- 8. Istio - Sail (Kubernetes- Helmsman or ship’s pilot) Envoy (High perfomance distributed proxy)
- 9. Istio Control Plane ISTIO- A ROBUST SERVICE MESH FOR MICROSERVICES Istio Pilot Istio Mixer Istio Auth Pod Container Service A Envoy Proxy Pod Container Service A Envoy Proxy Pod Container Service A Envoy ProxyIstio Data Plane ● service discovery ● load balancing ● TLS termination ● HTTP/2 & gRPC proxying, ● circuit breakers, ● health checks, ● staged rollouts fault injection ● rich metrics. ● access control ● usage policies ● telemetry collection ● traffic mgmt ● discovery ● authentication ● policy enforcement ● Id & credentials
- 10. LAB: DETECTING ANDPREVENTING ISSUES IN DISTRIBUTED APPS WITH ISTIO
- 11. GOAL FOR LAB Inthis lab you will learn: ● How to install Istio onto OpenShift Container Platform ● How to deploy apps with sidecar proxies ● How to generate and visualize deep metrics for apps ● How to alter routing dynamically ● How to inject faults for testing ● How to do rate limiting ● How Istio implements circuit breaking and distributed tracing
- 12. LAB: DETECTING ANDPREVENTING ISSUES IN DISTRIBUTED APPS WITH ISTIO SCENARIO 7 PREVENT AND DETECT ISSUES IN A DISTRIBUTED SYSTEM
- 13.
- 14. RESULT OF LAB Inthis lab you learned: ● How to install Istio onto OpenShift Container Platform ● How to deploy apps with sidecar proxies ● How to generate and visualize deep metrics for apps ● How to alter routing dynamically ● How to inject faults for testing ● How to do rate limiting ● How Istio implements circuit breaking and distributed tracing ● Use cases for service mesh
- 15. MICROSERVICES 4.0? Service Microservice Function f() > Single Purpose > Stateless > Independently Scalable > Automated > Single Action > Event-sourced > Ephemeral > Autonomous > Loosely-coupled
- 16. SERVERLESS PROJECTS /SERVICES 28 http://funcatron.org CLOUD FUNCTIONS BETA serverless-docker
- 17.
Editor's Notes
- #4 Why do we want to architect apps from a distributed services standpoint: The benefits, we hope to achieve are primary related to: Performance Reliability Resiliency Extensibility Availability Robustness Often application architectures must write code to facilitate these elements rather then just concentrating on core business logic.
- #5 Why is it important to ensure robustness among these distributed services elements? Because of this list of fallacies in Distributed Computing environments: The network is reliable. Latency is zero. Bandwidth is infinite. The network is secure. Topology doesn't change. There is one administrator. Transport cost is zero. The network is homogeneous. We typically can not trust that these items will be available in our environments….
- #6 Therefore, applications must deal with: Unpredictable failure modes End-to-end application correctness System degradation Topology changes Elastic/ephemeral/transient resources or resources that simply go away… One can argue that a highly available cluster environment, such as OpenShift running at scale would help mitigate many of these issues….? What do you all think…? Depending on the specific context, I’m good with this though… We’ve seen how health-checks work today… We understand how OCP can schedule multiple pods within the cluster adding resiliency to our individual micro-services. We’ve seen pipeline automation for CI/CD and testing scenarios complete with advanced deployment techniques -- again, working together to ensure robustness So, how can we make this environment even more resilient….?
- #7 Entire suites of frameworks were built to help developers address these resilience concerns… (e.g. netflix oss) Netflix OSS has been the de facto way of building these supporting services for running microservices which required a number of standalone services to maintain and operate as well as boilerplate code that is required to talk to these services Netflix OSS while a great collection of frameworks and libraries was build a couple of years ago for the VM-based AWS infrastructure. Fast forward to today, many of the services that didn’t exist at that time for the VMs and Netflix had to invent them, are provided as base services in container platforms
- #8 However, recognizing that there is too much infrastructure in our business logic… what other solutions can we turn too?
- #14 entire suites of frameworks so for every language/framework combination, you need... service discovery retries timeouts load balancing bulk heading circuit breaking rate limiting built to help developers address these resilience concerns (e.g. netflix oss) adaptive routing deadlines back pressure outlier detection health checking traffic shaping request shadowing edge/dmz routing surgical / fine / per-request routing A/B testing rollout dark launches fault injection stats, metric collection observability
- #15 Originally built at Lyft, Envoy is a high performance C++ distributed proxy designed for single services and applications, as well as a for large scale microservice “service mesh” architectures. Built on the learnings of solutions such as NGINX, HAProxy, hardware load balancers, and cloud load balancers. Envoy runs alongside every application (Pod -- the concept of a side-car) and abstracts the network by providing common features in a platform-agnostic manner. When all service traffic in an infrastructure flows via an Envoy mesh, it becomes easy to visualize problem areas via consistent observability, and tune overall performance. As a service mesh grows in size and complexity, it can become harder to understand and manage. Istio adds requirements including: discovery, load balancing, failure recovery, metrics, and monitoring, and often more complex operational requirements such as A/B testing, canary releases, rate limiting, access control, and end-to-end authentication. Traffic Management. Control the flow of traffic and API calls between services, make calls more reliable, and make the network more robust in the face of adverse conditions. Observability. Gain understanding of the dependencies between services and the nature and flow of traffic between them, providing the ability to quickly identify issues. Policy Enforcement. Apply organizational policy to the interaction between services, ensure access policies are enforced and resources are fairly distributed among consumers. Policy changes are made by configuring the mesh, not by changing application code. Service Identity and Security. Provide services in the mesh with a verifiable identity and provide the ability to protect service traffic as it flows over networks of varying degrees of trustability. In addition to these behaviors, Istio is designed for extensibility to meet diverse deployment needs: Platform Support. Istio is designed to run in a variety of environments including ones that span Cloud, on-premise, Kubernetes, Mesos etc. We’re initially focused on Kubernetes but are working to support other environments soon. Integration and Customization. The policy enforcement component can be extended and customized to integrate with existing solutions for ACLs, logging, monitoring, quotas, auditing and more. The goal: Is that these capabilities greatly decrease the coupling between application code, the underlying platform, and policy. This decreased coupling not only makes services easier to implement, but also makes it simpler for operators to move application deployments between environments or to new policy schemes. Applications become inherently more portable as a result. ***Istio currently only supports service deployment on Kubernetes, though other environments will be supported in future versions.
- #16 entire suites of frameworks so for every language/framework combination, you need... service discovery retries timeouts load balancing bulk heading circuit breaking rate limiting built to help developers address these resilience concerns (e.g. netflix oss) adaptive routing deadlines back pressure outlier detection health checking traffic shaping request shadowing edge/dmz routing surgical / fine / per-request routing A/B testing rollout dark launches fault injection stats, metric collection observability
- #18 Earlier this year Google, IBM and Lyft announced the Istio project, which aims to make it easier to develop and connect AND manage our complex and distributed microservices applications. First generation microservices were mainly A DIY effort - you had to do a lot of defensive programming in the app itself. Istio basically delivers on the “microservices 2.0” idea of minimizing requirements on developers for dealing with the distributed nature of their apps. In this diagram from Christian Posta, chief architect for cloud app dev at red hat, you can roughly see how this translates to apps - it pushes the real magic behind distributed apps like routing, rate limiting or circuit breaking into lower networking layers, out of reach and out of control of app developers. It also means that the service mesh benefits all applications running on it, regardless of programming language and communication protocol - think databases or IoT binary streaming apps.
- #19 An Istio service mesh is logically split into a data plane and a control plane. The data plane is composed of a set of intelligent proxies (Envoy) deployed as sidecars that mediate and control all network communication between microservices. The control plane is responsible for managing and configuring proxies to route traffic, as well as enforcing policies at runtime. The following diagram shows the different components that make up each plane: ################## To give you an idea of what this looks like, here’s a rough diagram of the components at play. Istio creates a cross-cutting platform-level service mesh to address common microservices architecture concerns, and there are a lot of them: communication, load balancing, traffic routing, metrics, quotas, authentication, rate limiting, circuit breakers, timeouts, automatic retries, and on and on.. Things that developers and operations have to deal with today. It does this by injecting we are called side car proxies for each service, acting as a frontend to the service and managing traffic to it according to policy. As services come and go, their presence, absence, or general health are tracked by The control plane components and traffic is shaped accordingly. Red Hat customers and the greater OpenShift and Kubernetes communities will benefit from this platform level support for microservice architectures, so stay tuned as we work to bring a developer preview of this technology by Red Hat Summit in May and then later on in the year fully integrate it into OpenShift and the RHOAR getting started experience.
- #20 To give you an idea of what this looks like, here’s a rough diagram of the components at play. Istio creates a cross-cutting platform-level service mesh to address common microservices architecture concerns, and there are a lot of them: communication, load balancing, traffic routing, metrics, quotas, authentication, rate limiting, circuit breakers, timeouts, automatic retries, and on and on.. Things that developers and operations have to deal with today. It does this by injecting we are called side car proxies for each service, acting as a frontend to the service and managing traffic to it according to policy. As services come and go, their presence, absence, or general health are tracked by The control plane components and traffic is shaped accordingly. Red Hat customers and the greater OpenShift and Kubernetes communities will benefit from this platform level support for microservice architectures, so stay tuned as we work to bring a developer preview of this technology by Red Hat Summit in May and then later on in the year fully integrate it into OpenShift and the RHOAR getting started experience.
- #23 We’re going to use a sample app within the Istio framework to run through these activities…. ################ To give you an idea of what this looks like, here’s a rough diagram of the components at play. Istio creates a cross-cutting platform-level service mesh to address common microservices architecture concerns, and there are a lot of them: communication, load balancing, traffic routing, metrics, quotas, authentication, rate limiting, circuit breakers, timeouts, automatic retries, and on and on.. Things that developers and operations have to deal with today. It does this by injecting we are called side car proxies for each service, acting as a frontend to the service and managing traffic to it according to policy. As services come and go, their presence, absence, or general health are tracked by The control plane components and traffic is shaped accordingly. Red Hat customers and the greater OpenShift and Kubernetes communities will benefit from this platform level support for microservice architectures, so stay tuned as we work to bring a developer preview of this technology by Red Hat Summit in May and then later on in the year fully integrate it into OpenShift and the RHOAR getting started experience.
- #24 We’re going to use a sample app within the Istio framework to run through these activities….Here’s a diagram of the logical architecture for our last scenario.