Your SlideShare is downloading. ×
Distributedsystems 100912185813-phpapp01
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Distributedsystems 100912185813-phpapp01


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. gn si De emDistributed Systems st Sy d te bu scalability and high availability ri st DiRenato Lucindo - - @rlucindo
  • 2. Renato Lucindo Call me Lucindo (or Linus) gn 2002 - Bachelor Computer Science si 2007 - M.Sc. Computer Science (Combinatorial De Optimization) em st 7+ year developing Distributed Systems Sy d te bu ri st Di My default answer: "I dont know."
  • 3. Agenda gn Scalability si De em st High Availability Sy d te Problems bu ri st Di Tips and Tricks Learning More
  • 4. Distributed Systems Multiple computers that interact with each other over a gn network to achieve a common goal si Purpose De Scalability em st High availability Sy d te bu ri st Di source:
  • 5. Scalability System ability to handle gracefully a growing amount of gn work si De Scale up (vertical) em st Add resources to a single node Sy Improve existing code to handle more work d te bu Scale out (horizontal) ri Add more nodes to a system st Di Linear (or better) scalability
  • 6. Scalability - Vertical Add: CPU, Memory, Disks (bigger box) gn Handling more simultaneous: si Connections De Operations em st Users Sy Choose a good I/O and concurrency model d Non-blocking I/O te bu Asynchronous I/O ri Threads (single, pool, per-connection) st Di Event handling patterns (Reactor, Proactor, ...) Memory model? STM
  • 7. Scalability - Vertical Careful with numbers gn Requests per second si # of Connections De Simultaneous operations em st Event handling Sy Think front-end d Slow connections/clients te bu Its slower than other options ri In doubt, go async st Di Back-end Thread pool (thread per-connection) No events Process per-core
  • 8. Scalability - Horizontal Add nodes to handle more work gn Front-end si Straightforward De Stateless em st Back-end Sy Master/Slave(s) d Partitioning te bu DHT ri Volatile Index st Di
  • 9. Scalability - Horizontal Master/Slave gn Write on single Master si Read on Slaves (one or more) De Scales reads em st Sy d te bu ri st Di
  • 10. Scalability - Horizontal Partitioning (Sharding) gn Distribute dada across nodes si Generally involves data de-normalization De Where is some specific data? em st Master Index Sy Hash (DTH, Consistent Hashing) d Volatile Index te bu Joins done in application level ri NoSQL friendly st Di
  • 11. Scalability - Horizontal Volatile Index: build and maintain data index as cached gn information (all clients) si De em st Sy d te bu ri st Di
  • 12. High Availability "Processes, as well as people, die" gn si De Handle hardware and software failures em st Eliminate single point of failure Sy Redundancy d Failover te bu Replicas ri st Di
  • 13. High Availability - Failover/Redundancy gn si De em st Sy d te bu ri st Di
  • 14. High Availability - Replicas Two or more copies of same data gn Replica granularity si From node replica to "row" replica De Load balancing em st Write concurrency Sy Replica updates d Key for high availability and root of several problems te bu ri st Di
  • 15. ProblemsDi st ri bu te d Sy st em De si gn
  • 16. Di st ri bu te d Sy st em De Problems - CAP Theorem si gn
  • 17. Problems - CAP Theorem Consistency: all operations (reads/writes) yield a global gn consistent state si De Availability: all requests (on non-failed servers) must have em st a response Sy d Partition Tolerance: nodes may not be able te bu to communicate with each other. ri st Di Pick Two
  • 18. Problems - CAP Theorem C + A: network problems might stop the system gn si Examples: De Oracle RAC, IBM DB2 Parallel em st RDBMS (Master/Slave) Sy Google File System d HDFS (Hadoop) te bu ri st Di
  • 19. Problems - CAP Theorem C + P: clients cant always perform operations gn si Examples: De Distributed lock-systems: Chubby, ZooKeeper em st Paxos protocol (consensus) Sy BigTable, Hbase d Hypertable te bu MongoDB ri st Di
  • 20. Problems - CAP Theorem A + P: clients may read inconsistent (old or undone) data gn si Examples: De Amazon Dynamo em st Cassandra Sy Voldemort d CouchDB te bu Riak ri Caches st Di
  • 21. Problem with CAP Theorem In practice, C + A and C + P systems are the same. gn C + A: not tolerant of network partitions si C + P: not available when a network partition occurs De Big problem: network partition em st Not so big (how often does it happens?) Sy Pick two d Availability te bu Consistency ri The forgotten: Latency st Di Or, how long the system waits before considering a partitioned network?
  • 22. Problems - Real WorldEvery component may fail: gn Network failure si Hardware failure De Electricity em st Natural disasters Sy Code failure d te bu ri st Di
  • 23. Tips & TricksDi st ri bu te d Sy st em De si gn
  • 24. Tips & Tricks - Pyramid Capacity (connections, operations, ...) Pyramid gn si De em st Sy d te bu ri st Di
  • 25. Tips & Tricks - Reply Fast FAIL Fast gn Break complex requests into smaller ones si Use timeouts De No transactions em st Be aware that a single slow operation or component can Sy generate contention d Self-denial attack te bu ri st Di
  • 26. Tips & Tricks - Cache Cache: component location, data, dns lookups, previous gn requests, etc si Use negative cache for failed requests (low expiration) De Dont rely on cache em st Your system must work with no cache Sy d te bu ri st Di
  • 27. Tips & Tricks - Queues Easy way to add asynchronous processing an decouple gn your system. si De em st Sy d te bu ri st Di
  • 28. Di st ri bu te d Tips & Tricks - DNS Sy st em De si gn
  • 29. Tips & Tricks - Logs Log everything gn Use several log levels si On every log message De User em st Request host Sy Component involved d Version te bu Filename and line ri If log level not enabled do not process log message st Di Avoid lookup calls (gettimeofday)
  • 30. Tips & Tricks - Domino Effect Make sure your load balancer wont overload components gn User smart algorithms si Load Balance De Resource Allocation em st Sy d te bu ri st Di
  • 31. Tips & Tricks - (Zero) Configuration No configuration files gn Use good defaults si Auto-discovery (multicast, gossip, ...) De Make everything configurable em st Administrative command Sy No need to stop for changes d Automatic self adjusts when possible te bu ri st Di
  • 32. Tips & Tricks - STOP Test With your system under load: kill -STOP <component> gn si De em st Sy d te bu ri st Di
  • 33. Tips & Tricks - Know your tools load average (uptime) gn stats tools si vmstat De iostat em st mpstat Sy tcpstat, tcprstat, etc d tcpdump, nc, netstat te bu tunning ri /proc/net/* st Di ulimit sysctl oprofile debuging tools (gdb, valgrind) ...
  • 34. Tips & Tricks - Count Count everything gn Connections si Operations De Failures em st Successes Sy Request times (granularity) d Total, average, standard deviation te bu Monitor counters ri st Di
  • 35. Tips & Tricks - Stability Patterns Use Timeouts gn Circuit Breaker si Bulkheads De Steady State em st Fail Fast Sy Handshaking d Test Harness te bu Decoupling Middleware ri st Di
  • 36. Tips & Tricks - Dont Panic! gn si De em st Sy d te bu ri st Di
  • 37. Learning More - BooksTCP/IP Illustrated, Vol. 1: The Protocols gn si De em st Sy d te bu ri st Di
  • 38. Learning More - BooksUnix Network Programming, Vol. 1: The Sockets Networking gn si De em st Sy d te bu ri st Di
  • 39. Learning More - BooksPattern Oriented Software Architecture, Vol. 2 gn si De em st Sy d te bu ri st Di
  • 40. Learning More - BooksRelease It! gn si De em st Sy d te bu ri st Di
  • 41. Learning More - Papers The Google File System gn Bigtable: A Distributed Storage System for Structured Data si Dynamo: Amazons Highly Available Key-Value Store De PNUTS: Yahoo!’s Hosted Data Serving Platform em st MapReduce: Simplified Data Processing on Large Clusters Sy d Towards robust distributed systems te bu Brewers conjecture and the feasibility of consistent, ri available, partition-tolerant web services st Di BASE: An Acid Alternative Looking up data in P2P systems
  • 42. gn si De emThanks!!! Questions? st Sy d te - @rlucindo ri st Di