Cloud Design Pattern part2
•Download as PPTX, PDF•
6 likes•1,463 views
Cloud Design Patterns for Availability
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Cloud Design Pattern part2
Similar to Cloud Design Pattern part2 (20)
Recently uploaded
Recently uploaded (20)
Cloud Design Pattern part2
- 1. Cloud Design Patterns – part 2 Masashi Narumoto AzureCAT patterns & practices Microsoft
- 2. Agenda
- 3. Retry problem – access to resource fails
- 5. Service Retry capabilities Policy configuration Scope Telemetry features AzureStorage Native in client Programmatic Client and individual operations TraceSource SQL Database with Entity Framework Native in client Programmatic Global per AppDomain None SQL Database with ADO.NET Topaz* Declarative and programmatic Single statements or blocks of code Custom Service Bus Native in client Programmatic Namespace Manager, Messaging Factory, and Client ETW Cache Native in client Programmatic Client TextWriter DocumentDB Native in service Non-configurable Global TraceSource Search Topaz* (with custom detection strategy) Declarative and programmatic Blocks of code Custom Active Directory Topaz* (with custom detection strategy) Declarative and programmatic Blocks of code Custom Retry solution
- 7. Circuit Breaker problem - remote service causes cascading failure
- 10. Queue Based Load Leveling problem Surge in requests exhaust resources X Web Site
- 11. Queue Based Load Leveling solution Web site
- 13. Throttling problem - surge in workload
- 16. Leader Election problem – control distribution
- 19. Valet Key problem – protect storage
- 22. ✔ ✔ ✔ ✔ ✔ ✔
- 24. Index Table problem Lack of 2ndary index
- 25. Index Table solution Pros: - Query is fast since the new table has all data Cons: - Table size becomes large - Data consistency issue
- 26. Index Table solution Pros: - Table size is small - Less consistency issues Cons: - 2 step lookup makes query slow
- 27. Index Table solution Pros: - Query is fast - Table size is small Cons: - Depends heavily on Data access pattern
- 30. Static Content Hosting problem – storing static content cspkg Web site
- 31. Static Content Hosting solution cspkg cspkg Web site CDN/Blob
Editor's Notes
- Today, I’m going to talk about 9 patterns that I think most useful or important to understand. Doing 9 is still challenge but let’s see how far we can go. A little icon by each pattern represents the category. Sharding belongs to scalability, MV, ES, CQRS belongs to data mgmt. etc.
- Retry pattern looks very simple but it’s actually the most complicated one to implement. I’ll show you why. This is also very important pattern since almost 50% of the system trouble is caused by poor implementation of this pattern. In the cloud, you’ll see lots more transient faults than on-prem because of NW and HW
- Solution may look simple. Just retry the operation when it fails. But the question is when and how many times? How long the interval should be? Often, we see fixed settings like 1 sec, 3 times across all operations but it’s not good for 2 reasons. First, you don’t consider the E2E latency requirement. If E2E transaction should be done within 2 sec, 3 x 1 sec could take longer than that. Second, if all operations retry with the same interval, it’ll hammer the remote service by sending 100s of requests at the same time. So the interval should be randomized as well as delayed.
- SQL DB, Azure Search, AAD doesn’t provide retry library. Some you can configure policy through program, configuration file or both. Scope of the policy is different per service as well.
- You need to figure if the error is transient or not then retry only Transient errors. Use short linear interval for interactive use case, exponential back-off for batch use case. One of the anti-patterns is a cascading retry where both outer and inner method does retry. Most of Azure service offer built-in retry mechanism except for SQL DB, AAD and Azure Search. Increasing # of retry indicates something going on behind the scene. Log and analyze retry operations
- Call to a remote service could fail for many reasons. Server internal error, Network cloud be down or a network device cloud be down. The question is if it’s transient or non-transient fault. That’s what matters most in this context. If it’s transient fault, you should retry the operation a few times and see if it goes through as we already discussed. But if the fault is non-transient, you don’t want to retry. You don’t even want to make the first attempt from the get go. Since you know it’s going to fail. The problem is not just wasting your time by sending request that’s likely to fail. Bigger problem is that the failure is going to be cascading. Remote service call takes resources such as memory, threads and network connections. If there’re millions of calls, it’ll eat up all available resources and not just this transaction but other part of the system is also going to fail. Also by keep hummering the remote service, it can’t recover from failure. So you don’t want to keep calling the service while it’s down. You want to do something else. This is where circuit breaker comes into play.
- It acts as a proxy to the remote service. And it’s also a state machine with three states, closed, open and half-open. We’re using a circuit breaker as an analogy. So closed is the normal state, open is a fault state. Half-open is where we examine the service to see if it comes back normal. When it’s closed, all requests are going through to the service. It counts number of recent failures. If it exceeds a specified threshold, it trips to the open state. In the open state, the requests fail immediately without any attempt, and exceptions are returned to the client. When it’s getting into the open state, it starts a timer. When the timer expires, it moves to half-open state. In the half-open state, limited number of request are going through. If a specified number of consecutive requests are successful, it assumes that the problem was fixed and it resets the state to closed. It the request fails, it goes back to open state, another timer kicks in and repeat the same step.
- 1. Exception handling must be app specific. You may want to degrade its functionality, use the cached data, invoke alternative operations. 2. Instead of timer, circuit breaker may periodically ping the remote service to see it has become available again. 3. If the recovery time is extremely variable, it may be better to let an administrator to manually close the breaker instead of a timer. Similarly an administrator can force a breaker into the open state if it’s temporarily unavailable. 4. If the circuit breaker protects the remote DB which is partitioned into multiple shards., one shard may be fully accessible while the another is experiencing an issue. 5. Sometimes a response can contain enough information for the circuit breaker to trip immediately to open state. For instance, HTTP 503 “service unavailable” can include additional info such as anticipated duration of the delay.
- This is a simple but powerful design pattern. No matter how many instance you scale out the web site instance, DB can’t handle the request. For instance, P3 can handle only 735 rpm. Once you go beyond that, it can reject your request for next 10 seconds. How can you avoid the throttling?
- Solution is to insert a queue in a middle. Use a queue as a buffer then the backend worker can process them at own pace.
- 1. Like every 10 seconds, get 50 msg and process them. 2. To identify right # of queue, resources, you need to performance test with expected MAX workload 3. If a client need a response, use reply queue to send back the result 4. By adding a queue, the e2e latency will be increased
- You can’t precisely predict the workload. It fluctuates due to many reasons. In multi-tenant system, aggregated volume of requests may go beyond your imagination. In any case, once it goes beyond the capacity, the system will start suffering from poor performance. One way to deal with this problem is auto-scaling, however it takes time to provision additional services. It also incur additional expenses.
- Idea here is to allow applications to use resources only up to some soft limit, when the limit is reached, throttle them. There’re several strategies to implement this idea. 1. Disable the non-critical functionality so that essential service can keep running with sufficient resources. The vertical dimension shows resource utilization such as memory, CPU, network etc. while the horizontal one shows the time. There’re three features A,B and C. At time T1, the total resource usage reaches the threshold. Among these three, Feature B is the least critical so it’s temporarily disabled while A and C continue running as normal. By disabling feature B, it will stop its resource consumption so A and C can make use of it. In other word, we’re re-allocating resource to the features in high priority. At time T2, resource use of these features diminishes so we can enable feature B again. You need to monitor the aggregated resource consumption and see if it reaches the threshold all the time. Other options are.. Simply reject the request from an individual user who’s making too many requests. Take facebook Graph API for example, it throttles when you make over 600 requests per 10 minutes. It requires metering each individuals. 3. Use queue as a buffer to requests and process them at your own pace or prioritize them (premium vs. standard users)
- 1. Throttling could make a significant impact on the entire system design. And there’re a number of ways to implement such as disabling non-critical features, rejecting request from individual user, load leveling using queue etc. In any case, it can’t be after thought. 2. In any strategies, it must be performed quickly. Detect the increase in activity and react and also after the load has eased, the system must revert back to original state. 3. Autoscaling and throttling are not mutually exclusive. They can be used together. Use throttling as a temporary measure. 4. If demand grows very quickly, even throttling may not be able to protect the system. Consider aggressive auto-scaling by maintaining larger reserve of capacity.
- This is a very common problem in the distributed system. There’re multiple nodes, one of them has to control the entire workflow. Be it split/shuffle in map reduce process or dispatch requests in elasticsearch data ingestion. In these cases, you need to select one node as a master. How can we do that?
- Essentially there’s 2 ways, either using algorithm or a shared resource. There’re a number of algorithm ranging from Bully, Ring to more sophisticated ones. Most simple algorithm is to pick the smallest number (instance ID) among all instances. Second way is use a distributed mutex. Here’s an example of using blob lease. The first node instance that acquires the lease is the leader.
- The process of electing the leader may fail, make it resilient by retrying the process. The elected leader may go down, replace it with a new leader when it happens like what we discussed in the previous slide. When you use distributed mutex like blob lease, it could be a SPOF. Be aware of that. If you turn on auto-scaling, the leader may be removed by that.
- When the clients need to access the data, the application takes care of it by fetching the data from storage and streaming it to the client. Or other way round, by reading uploaded data and store it to the storage. This is what we usually do. This approach requires lots of resources in the app such as compute capacity, memory, and network bandwidth. However the application in this context is just the intermediary isn’t it? It just receives data coming from the client and transfer it to the storage or other way round. So why don’t we bypass the app and then connect the client with the storage directly? Most of the data store have capability to handle upload and download of data directly without application intervention. This approach is useful to maximize performance, scalability and minimize the cost. All is good! However, there’s one drawback of this approach. the web site is no longer able to manage the security of the data. Because app acts like gatekeeper but we bypass it. So now nobody validates the access for you. Is there any way to provide data directly from the storage and secure the access at the same time? This is where valet key pattern comes in.
- Solution is to restrict access to the data store by giving client a key that the data store can validate. This key is usually referred to as a valet key. As the name suggests, Valet key is used in the valet parking. It gives only restricted access to the car, You can’t do anything but open the door and start engine . You can’t open the trunk or dashboard. We use the same concept here. The valet key normally gives only time-limited access to a specific resources and allows only predefined operations like read or write. The key can be configured to restrict the access to limited scope of the data. For instance, for the blob, it could give access to only specific container or specific items in a container. The key also can be invalidated by the application at any time. For instance, if the data download operation is completed, the client can tell the application that it’s completed and the key is immediately invalidated to make it one time thing.
- Limit the time period and scope of the resource as tightly as possible. If it’s one time thing, don’t give it one hour time window. 2. Similarly you should give users only required level of access, for uploading scenario, give them only write access. And vice versa. 3. It is usually not possible to limits the size of the data, or the number of times to access the data. There‘s a workaround by forcing client to notify application when one operation is complete. 5. Even the key provides restrict access, there’s a small chance that a malicious user gain the access to the key. To protect from malicious attack, it’s a good practice to validate the uploaded data before processing it. If it doesn’t conform to the specified schema, don’t process it. 6. The default of start time is normally the current server time but if the client time is bit behind of server the key may not yet be available at the time it’s handed off to the client. Clock skew problem. Ensure that the start time is a little earlier than the current server time.
- There’re 4 ways. 1. Create another table w/ index using Town or Name and all other fields
- Create a normalized index only table which has just indexes.
- This is option 1 and 2 combined. Create a partially normalized table with index and only frequently accessed fields.
- If majority of data access is using more than one keys, combine them by concatenating