Resilience is by design
•
29 likes•10,463 views
Resilience design pattern. What is resilience? What's different? and How can we use it. Let us use it to design E-commerce flush sales events. Second case is a Uber like App design.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Resilience is by design
Similar to Resilience is by design (20)
Recently uploaded
Recently uploaded (20)
Resilience is by design
- 2. About Me • Director, EHS • Principle Engineer, Yahoo • Sr. Engineering Manager, Yahoo • Solution Architecture, Yahoo • Yahoo Frontpage • Yahoo EC Shopping – VIP Box • Yahoo iOS movies • Yahoo Finance • Yahoo News • IT Head, Puma/Starlike • SAP IS-Retail I don’t know what I don’t know until I know it. Ask and learn. FB: https://www.facebook.com/im.young.yang
- 8. What’s different? • Fault tolerant • Disaster Recovery • High Availability • Stability • Reliability
- 10. Disaster Recovery A good disaster recovery solution is all about getting your business’s valuable data and operations out of the failed infrastructure and running again on new infrastructure. Disaster recovery is not about saving the infrastructure. Disaster recovery is about saving the business.
- 11. High Availability A high availability design means that the outage will be brief because it will not take long to redeploy the required component. Unlike replacing a tire with high availability the redeployment is completely automated following a component failure.
- 19. Reliability There are 3 sub characteristics: • Maturity : concerns frequency of failure of the software. • Fault tolerance : The ability of software to withstand (and recover) from component, or environmental, failure. • Recoverability : Ability to bring back a failed system to full operation, including data and network connections. - By ISO9126
- 20. DIE Hard 4
- 22. Reliability vs Resiliency Reliability is the outcome and resilience is the way you achieve the outcome. - By Microsoft Secure Blog
- 23. Take away
- 32. Resilience Design for flush sales Application/ Services Infrastructure Architecture • Queue • No-SQL: Key Value • Traffic splitter • Stress Testing • 3-round burst • Token • Stateless vs stateful • Fast Pass • Degrade gracefully • Fast Track for Payment • One man limited • Oversold / Overbooking
- 34. Queue - Mailbox • Async • Size Limited, overflow • 印花抽獎 • LuckyDog Infra
- 41. Architecture
- 42. Take Away – Queue Queue Order Order Order Order Order Order Order Benefits • Easy to implement • All requests are going the queue • Parallel create orders Drawbacks • Call back “users”? • Quickly finish < 10 seconds • Hard to get the crowd back if we have dropped orders. Infra
- 45. Challenges • A smart queue • It’s impossible to call back “users”. • Avoid sticky sections. • Keep stateless for the frontend. Can quickly launch machines and share the loads • A shared backend and can afford the capacity • Minimize states • # of total requests / second • # of limited requests to checkout • # of current inventory • # of planed inventory • 3 round burst
- 47. Shared Backend and No-SQL Key Value DB Why Redis • Very fast: 8~10W QPS • Simple • Sentinel: easy to implement HA • Key length size: 512MB • Key design: • Use colon to structure meaning • Eg. 20170411:VIPBOX:INVENTORY Infra
- 48. User Behavior
- 49. We though Users’ request will decrease 12000 12500 13000 13500 14000 14500 15000 15500 16000 16500 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Request per second Application/ Services
- 50. Lottery winning ratio design – based on QPS 0.000000 0.000500 0.001000 0.001500 0.002000 0.002500 0.003000 0.003500 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Lottery Winning Ratio Benefits • The winning ratio will increase after time passed. • Eventually, sold out. Drawbacks • Complex: based on previous numbers to adjust winning ratio. Applicati on/ Services
- 52. Users are not the enemy. But sometimes are. 100 110 120 130 140 150 160 170 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Hundreds Request per Second Wave 1 Wave 2 Wave 3 Wave 4 Applicati on/ Services
- 60. Key Take Away – 3-round burst • Use O(1) commands. Don’t Use O(n) commands. • PHP and Node.js are different. • You can’t expected Node.js callback “microsecond” delay time interval. • Use persistent connection to the redis server. Reduce the open connection overhead • Must execute loading tests. Architecture
- 64. Fast Track
- 65. Fast Track
- 66. One man limited
- 67. One man limited – Client side and Server side • Client side – Use token or cookie • Some flush sales design: if you did not win “lottery” ticket at first connect. You never have chances. => clean up your cookie and cache. • Benefits: easy to implement and scale. • Server side – Use Redis Server • Keep user id to record done-payment. • Drawbacks: need to change your checkout logics for this special case. Suggest don’t add this business constraints
- 69. CAP Theorem
- 70. CAP Theorem (定理) - pickup 2 Consistency Every read receives the most recent write or an error Availability Every request receives a (non- error) response – without guarantee that it contains the most recent write Partition tolerance The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes
- 75. Take away - resilience design for flush sales Application/ Services Infrastructure Architecture • Queue • No-SQL: Key Value • Traffic splitter • Stress Testing • 3-round burst • Token • Stateless vs stateful • Fast Pass • Degrade gracefully • Fast Track for Payment • One man limited • Oversold / Overbooking
- 78. [無言] 我訂了120份摩斯早餐！ - Part 1
- 79. [無言] 我訂了120份摩斯早餐！ - Part 2
- 80. [無言] 我訂了120份摩斯早餐！ - Part 3
- 87. How to fix it?
- 88. Resilience Design to avoid ”Users’ Retry” Application/ Services Infrastructure Architecture • Queue • Alter for slow query • Stress Testing • Circuit break • Event Centric Architecture • Pub / Sub • Failover • JSON Web Token (JWT) • Short term Token • Dedupe • Timeout • Network Detection • Fallback • Evergreen
- 93. Token Base vs Cookie Base Benefits • Stateless : backend don’t need to keep token records. • Scalable: server don’t need to consider O(n) • Decouple: app and security server can be separated. • Cross-Origin Resource Sharing (CORS) : cookie can’t cross domain • Mobile Ready Drawbacks • Size • Encode/Decode: don’t put sensitive data
- 94. Short term Token • “exp”: expiration time. • Setup a time period to expire. • If it is expired, api redirect user to login page. • Avoid to “replay” attack: add deviceID and from address. Application/ Services
- 95. Dedupe -How can we quickly do it? • Design unique key => use JWT signature • Set Key and EXPIRE in Redis => if EXISTS(Key), ask and confirm to the user. • Time Slot Window => use JWT expired value • Compare data Data Table Time Slot Window For example: 10 mins Benefits • Avoid to scan all data • Use No-SQL quickly check • User confirm to make order in such short time slot window. Be aware • Key length -> max 64 bytes for SHA256. • Can’t use memcached 250 chars. • Atomically data table and redis key Redis: JWT + EXPIRE
- 96. App Time Out • App Timeout < API timeout. • API timeout should be short. 10 seconds is enough. • Remember to disable send button. Application/ Services
- 99. Fallback
- 100. Fallback
- 101. Evergreen • Evergreen News: A post last for years, a long-lasting content page. • 如果你看到 Yahoo 首頁傳說中的熊貓，恭喜你進入 fallback mode. Application/ Services
- 102. Take away for Application / Services • JSON Web Token (JWT) • Short term Token • Dedupe • Timeout • Network Detection • Fallback • Evergreen Application/ Services
- 109. Failover Need to Consider • How long is your down time? • What is your business SLA? Eg. Trip requests • Do we need to process the queue data? 而且在我去車站途中，一直說沒車的app開始顯示有車 （崩潰） 在我離開後還是一直有計程車要進去那條巷子 Architecture
- 110. Take Away • Circuit break • Event Centric Architecture • Pub / Sub • Failover Architecture
- 111. Queue Benefits • Asynchronous: Queue it now, run it later. • Decoupling: Separates application logic. • Resilience: Won't take down your whole application if part of it fails. • Redundancy: Can retry jobs if they fail. • Guarantees: Makes sure that jobs will be processed. • Scalable: Many workers can process individual jobs in a queue. Infra
- 112. Queue Infra
- 113. Infra
- 114. Infra
- 117. Stress Testing Infra • Duration • Volume • Don’t screw up your sales report
- 118. Take Away • Queue • Alter for slow query • Stress Testing Infra
- 119. Take Away - Resilience Design to avoid ”Users’ Retry” Application/ Services Infrastructure Architecture • Queue • Alter for slow query • Stress Testing • Circuit break • Event Centric Architecture • Pub / Sub • Failover • JSON Web Token (JWT) • Short term Token • Dedupe • Timeout • Network Detection • Fallback • Evergreen
- 122. Design Principles Prevention • Monitor and Alert • No single point of failure • Fault isolation • Prevent propagation of the failure Cure • How do we achieve velocity with confidence?
- 123. Always ask these 3 questions • How critical is the component? • How likely is the component to fail? • How expensive is it to make the component fault tolerant or high available?
- 131. Take Away – Common Resilience Design Patterns Application/ Services Infrastructure Architecture Application / Services • Micro-services • Service level objectives • Fallback • Every green • Critical paths • Data: stateless, persistent • Archive strategy • Degrade services • Postmortem Infrastructure • Network • RAID 0, 1, 5, 6, 0+1, 10 • Cluster • Service discovery • Multi-Data Centers • Monitoring / Alert / Log • Traffic splitter • Load Testing Architecture • Detection • Recovery Limited Retry Reconnect Restart Rollback Failover colo Supervision • Treatment Hotfix • Mitigation avoid chain of failure Bulkheads Circuit Breaker Traffic splitter Bucket Fallback Every green Default value
- 132. Appendix
- 133. References - 1 • https://conferences.oreilly.com/velocity/velocityny2013/public/schedule/detail/31784 • https://www.youtube.com/watch?v=NjPVJ7ZKLnY • https://www.slideshare.net/ufried/resilience-reloaded-more-resilience-patterns • http://www.weibull.com/hotwire/issue26/relbasics26.htm • http://www.pbenson.net/2014/02/the-difference-between-fault-tolerance-high-availability-disaster- recovery/ • http://www.sqa.net/iso9126.html • http://softwareengineering.stackexchange.com/questions/158054/stability-vs-reliability • https://blogs.sap.com/2012/08/22/resiliency-vs-high-availability/ • https://gotocon.com/dl/goto-cph-2015/slides/JonasBonr_WithoutResilienceNothingElseMatters.pdf • http://www.metropolismag.com/Point-of-View/March-2013/Toward-Resilient-Architectures-1-Biology- Lessons/ • https://www.slideshare.net/InfoQ/breaking-bad-at-netflix-building-failure-as-a-service • http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf • https://www.youtube.com/watch?v=CZ3wIuvmHeM, Mastering Chaos • https://github.com/spinnaker/spinnaker • http://rhelblog.redhat.com/tag/redblack-deployment-model/ • http://techblog.netflix.com/2013/08/deploying-netflix-api.html • http://news.blogs.cnn.com/2011/07/26/owners-of-last-house-standing-part-of-peninsulas- comeback/comment-page-2/ • https://www.youtube.com/watch?v=IkPb15FfuQU
- 134. References - 2 • https://www.youtube.com/watch?v=IkPb15FfuQU • https://research.google.com/archive/chubby.html • https://www.slideshare.net/mongodb/everything-you-need-to-know-about-sharding • Scaling MySQL: https://www.youtube.com/watch?v=44tRhxGYXTY • https://www.moedict.tw/%E7%96%8A%E5%BA%8A%E6%9E%B6%E5%B1%8B • http://www.essi.upc.edu/~webgessi/publicacions/SMEF%2704-ISO-QualityModels.pdf • https://www.vprofx.com/blog/live-free-or-die-hard • https://ics-cert.us-cert.gov/sites/default/files/ICSJWG- Archive/QNL_MAR_16/reliability%20and%20resilience%20pdf.pdf • https://blogs.microsoft.com/microsoftsecure/2014/03/24/reliability-series-1-reliability-vs-resilience/ • https://ics-cert.us-cert.gov/sites/default/files/ICSJWG- Archive/QNL_MAR_16/reliability%20and%20resilience%20pdf.pdf • https://www.ptt.cc/bbs/Lifeismoney/M.1439174326.A.283.html • https://imgflip.com/i/1o7ahp • https://www.theregister.co.uk/2013/03/18/servers_pets_or_cattle_cern/ • https://www.mobile01.com/topicdetail.php?f=634&t=3814613 • http://www.spectatornews.com/opinion/2017/04/18/cost-benefit-of-overbooking-flights/ • https://en.wikipedia.org/wiki/CAP_theorem • http://robertgreiner.com/2014/08/cap-theorem-revisited/ • https://www.ptt.cc/bbs/StupidClown/M.1492074908.A.294.html • https://en.wikipedia.org/wiki/JSON_Web_Token • https://auth0.com/blog/cookies-vs-tokens-definitive-guide/
- 135. References - 3 • https://en.wikipedia.org/wiki/JSON_Web_Token • https://auth0.com/blog/cookies-vs-tokens-definitive-guide/ • http://morgan.dartmouth.edu/Docs/sas92/support.sas.com/documentation/cdl/en/bisecag/61133/ HTML/default/a003172284.htm • https://www.slideshare.net/IanBarber/zeromq-is-the-answer?qid=d40c9525-4824-45c4-aebc- 33d826d52194&v=&b=&from_search=1 • http://androidarena.co.in/tag/internet-check/ • http://www.javaworld.com/article/2824163/application-performance/stability-patterns-applied-in- a-restful-architecture.html?page=2 • https://pttread.com/t/m-1485244599-a-dd4 • https://msdn.microsoft.com/en-us/library/bb924374.aspx • http://gatling.io/docs/2.2/general/reports/ • https://en.wikipedia.org/wiki/Enterprise_service_bus • https://www.slideshare.net/IanBarber/zeromq-is-the-answer?qid=d40c9525-4824-45c4-aebc- 33d826d52194&v=&b=&from_search=1 • http://gatling.io/docs/2.2/general/reports/ • https://msdn.microsoft.com/en-us/library/bb924374.aspx
- 137. RAID
- 141. RAID 6: Independent Data Disks with Two Independent Parity Schemes Minimum 4, allow 2 disks failure
- 143. RAID 60
- 145. Cluster
- 146. Cluster vs Grid
- 147. Grid (1990s ~ 2007) Grid computing is the collection of computer resources from multiple locations to reach a common goal. • Loosely coupled • Heterogeneous network • Geographically dispersed
- 150. Clustering • Master – slave: problem: Vertical scaling: storage space • Shard- need to consider more.. Very complex • Shard Clustering: MongoDB and Redis • Master – master, or multi-masters • Masterless • P2P network • Ring • Gossip protocol O(1) routing, Multi-hop routing
- 151. Multi-Masters
- 153. Client Side (service consumer tenets) • Design for failure • Plan to be throttled • Plan to retry (with exponential back off) • Degrade gracefully • Cache when appropriate
- 154. Service Provider Tenets • Publish standard matrix • Publish logs for aggregation • Be prepared to throttle • Implementation details are private • Maintain backwards compatibility
- 155. Failover vs Failback vs Fallback • FAILOVER - switching to a secondary system when a failure exists in an application, network, etc • FAILBACK - restoring the system to a previous working state • FALLBACK - choosing an alternate service
- 158. Fallacies of distributed computing 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous • Must have error-handling on networking errors. • Minimize the communication cost as bandwidth, size, latency, frequencies, etc.
- 160. Bulkhead
- 161. Distributed Data • Merge conflict data? • Time Vector • Last modified win
- 162. Application/services • Micro-services • Service level objectives • Fallback • Every green • Critical paths • Data: stateless, persistent • Archive strategy • Degrade services • Postmortem
- 163. Infrastructure • Network • RAID 0, 1, 5, 6, 0+1, 10 • Cluster • Service discovery • Multi-Data Centers (Multi-colos) • Monitoring / Alert / Log • Traffic splitter • Load Testing
- 164. Architecture • Detection • Recovery • Limited Retry / Reconnect / Restart • Rollback • Failover colo • Supervision • Treatment • Hotfix • Mitigation – avoid chain of failure • Bulkheads • Circuit Breaker • Traffic splitter / Bucket management • Fallback / Every green / default value
- 165. ACID Model (CA) • Atomic • All operations in a transaction succeed or every operation is rolled back. • Consistent • On the completion of a transaction, the database is structurally sound. • Isolated • Transactions do not contend with one another. Contentious access to data is moderated by the database so that transactions appear to run sequentially. • Durable • The results of applying a transaction are permanent, even in the presence of failures.
- 166. BASE Model (AP) • Basically Available: • The database appears to work most of the time. • Soft state • Stores don’t have to be write-consistent, nor do different replicas have to be mutually consistent all the time. • Eventually consistent • Stores exhibit consistency at some later point (e.g., lazily at read time).
- 167. Challenges • CAP theorem requires to choose between availability and ACID-style consistency. • 2 phases commit should be avoided • Need a distributed transaction manager to support 2PC • All distributed systems such Database and queue must support 2PC • Most modern technologies, such as NoSQL database, does not have 2PC