Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Design for Scale / Surge 2010

on

  • 3,368 views

Christopher Brown's surgecon2010 talk on resilient, scalable systems based on his work on Amazon's EC2 and the Opscode Platform.

Christopher Brown's surgecon2010 talk on resilient, scalable systems based on his work on Amazon's EC2 and the Opscode Platform.

Statistics

Views

Total Views
3,368
Views on SlideShare
3,360
Embed Views
8

Actions

Likes
6
Downloads
38
Comments
1

2 Embeds 8

http://someguysblog.com 7
http://zeaffiliate.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Design for Scale / Surge 2010 Design for Scale / Surge 2010 Presentation Transcript

  • Design for Scale Christopher Brown VP, Engineering ‣ cb@opscode.com ‣ @skeptomai ‣ www.opscode.com Copyright © 2010 Opscode, Inc - All Rights Reserved 1
  • Who am I? Copyright © 2010 Opscode, Inc - All Rights Reserved 2
  • • Amazon EC2 Who am I? Copyright © 2010 Opscode, Inc - All Rights Reserved 2
  • • Amazon EC2 • Microsoft Edge Computing Network Who am I? Copyright © 2010 Opscode, Inc - All Rights Reserved 2
  • • Amazon EC2 • Microsoft Edge Computing Network • Opscode Who am I? Copyright © 2010 Opscode, Inc - All Rights Reserved 2
  • Google, Amazon, Microsoft built their own tools
  • P almost everyone else is here... ... inexperienced or poorly equipped for the world in which we now operate. Copyright © 2010 Opscode, Inc. – Confidential – Do Not Redistribute 4
  • The Method http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/
  • The Method Bootstrapping http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/
  • The Method Bootstrapping http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/
  • The Method Bootstrapping Configuration http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/
  • The Method Bootstrapping Configuration http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/
  • The Method Bootstrapping Configuration Command & Control http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/
  • The Method Bootstrapping Configuration Command & Control Nanite! http://www.flickr.com/photos/wonderlane/2090966628/sizes/l/
  • Got it? Copyright © 2010 Opscode, Inc - All Rights Reserved 6
  • Defining the cloud is like this... Got it? Copyright © 2010 Opscode, Inc - All Rights Reserved 6
  • Origin Myth of EC2 Copyright © 2010 Opscode, Inc - All Rights Reserved 7
  • Origin Myth of EC2 Copyright © 2010 Opscode, Inc - All Rights Reserved 7
  • Origin Myth of EC2 Copyright © 2010 Opscode, Inc - All Rights Reserved 7
  • Origin Myth of EC2 Copyright © 2010 Opscode, Inc - All Rights Reserved 7
  • Origin Myth of EC2 Copyright © 2010 Opscode, Inc - All Rights Reserved 7
  • Dynamism
  • Dynamism ...not about excess capacity...
  • Dynamism
  • Dynamism • Disintermediation • Developers can freely experiment
  • Dynamism • Disintermediation • Developers can freely experiment • Isolation • Applications safely co-exist
  • Dynamism • Disintermediation • Developers can freely experiment • Isolation • Applications safely co-exist • Utilization • Best use of expensive resources
  • Dynamism • Disintermediation • Developers can freely experiment • Isolation • Applications safely co-exist • Utilization • Best use of expensive resources This is what you are paying for
  • Scale
  • Scale You are not this BIG
  • Scale You are not this BIG
  • You are not that BIG • LAMP can scale on generic architecture • 2008 - Facebook has over 800 memcached servers, with 28 terabytes of RAM • 2010 - Github has 16 physical machines, 128 cores, 288 GB RAM • Don’t design for A Million Users • Ship early, Ship ugly, Ship often!
  • You are not that BIG • LAMP can scale on generic architecture • 2008 - Facebook has over 800 memcached servers, with 28 terabytes of RAM • 2010 - Github has 16 physical machines, 128 cores, 288 GB RAM • Don’t design for A Million Users • Ship early, Ship ugly, Ship often!
  • EC2 Design Principles • Minimize management footprint • Run in VMs just like customers. • Forced to analyze what must run in privileged space • “Harden everything” means separate network traffic inside the datacenter – http://www.flickr.com/photos/europedistrict/4058066840/ customers and management run there • True multi-tenancy - Customers run side- by-side • Design by Fight Club • "You are not a beautiful and unique snowflake“ • “On a large enough time line, the survival rate for everyone will drop to zero.” 
  • • Simple API, single unit of work • think of early Unix tools (MH) • Can compose with other APIs • Primitives • Does not define policy / coupling Customers will surprise you Copyright © 2010 Opscode, Inc - All Rights Reserved 13
  • APIs, Mashups Copyright © 2010 Opscode, Inc - All Rights Reserved 14
  • • Move complexity “up the stack” • Easier to debug • “Simple and Open” wins • OAuth, OpenID • ATOM, REST • Example: EC2 Metadata - Simplify HTTP http://www.flickr.com/photos/jfseesthings/4293062294/sizes/l/ Copyright © 2010 Opscode, Inc - All Rights Reserved 15
  • Cost
  • Cost • CapEx versus OpEx
  • Cost • CapEx versus OpEx • The Cloud is not “Cheaper”
  • Cost • CapEx versus OpEx • The Cloud is not “Cheaper” • Do you have money, time, or experience?
  • Cost • CapEx versus OpEx • The Cloud is not “Cheaper” • Do you have money, time, or experience? What are you willing to pay for?
  • Power Copyright © 2010 Opscode, Inc - All Rights Reserved 17
  • Power Copyright © 2010 Opscode, Inc - All Rights Reserved 17
  • Power Copyright © 2010 Opscode, Inc - All Rights Reserved 17
  • Nobody ever imagined a band of Orcs would steal a database table Charles Stross - Halting State
  • MTTF & MTTR Understanding how, when and why things fail is great ... but http://www.flickr.com/photos/dierken/948171048/sizes/z/
  • MTTF & MTTR Understanding how, when and why things fail is great ... but If your Mean Time to Recover exceeds the time value of your data, your business is DEAD http://www.flickr.com/photos/dierken/948171048/sizes/z/
  • Testing • Test with production-like dataset and performance • Don’t do “Design by Laptop” • A/B Testing • API versioning
  • Pull the Plug •Create test environment •Pull the plug •Document •Pull the plug again! http://www.flickr.com/photos/rosipaw/5033284534/sizes/m/in/photostream/
  • Pull the Plug •Create test environment •Pull the plug •Document •Pull the plug again! http://www.flickr.com/photos/rosipaw/5033284534/sizes/m/in/photostream/
  • Theo Morpheus vs
  • Theo Morpheus vs • Vertical vs Horizontal Scale • Availability • Reliability • 99% vs 99.x% per unit?
  • Theo Morpheus vs • Vertical vs Horizontal Scale • Availability • Reliability • 99% vs 99.x% per unit? Free your mind...
  • Theo Morpheus vs You are not Theo • Vertical vs Horizontal Scale •Availability • Reliability • 99% vs 99.x% per unit? Free your mind...
  • Theo Morpheus vs You are not Theo You’re probably not Morpheus either • Vertical vs Horizontal Scale •Availability • Reliability • 99% vs 99.x% per unit? Free your mind...
  • Theo Morpheus vs You are not Theo You’re probably not Morpheus either • Vertical vs Horizontal Scale •Availability • Reliability • 99% vs 99.x% per unit? Free your mind...
  • Availability • For a distributed system to be continuously available, every request received by a non-failing node in the system must result in a response. • “Read globally, Write locally" with inconsistent cache • Service Level Agreements, even (especially?) internally
  • Think Globally, Act Locally • Global but inconsistent aggregate view • Local action where data is authoritative • Autonomy • “Rightsizing” your failure domain http://www.flickr.com/photos/28634332@N05/3872137437/sizes/m/in/photostream/
  • Distributed Systems Design • Avoid execution caching • “Don’t lie, don’t retry” • Embrace failure • Don’t block the client • Avoid internal policy • Ensure the system makes forward progress
  • Embrace Failure • It’s OK to apologize • It’s better to completely fail for some users than penalize all of them • The Web is all about “Hit Refresh” Copyright © 2010 Opscode, Inc - All Rights Reserved 26
  • Apologize ...to Pat Helland
  • Admission Control • Distributed Throttling • Staged / Pipeline with back pressure • Measure scalability at each stage • Degraded performance • Make progress for admitted requests • At odds with “stateless” / session-less http://www.flickr.com/photos/jayneandd/4450623309/sizes/m/in/photostream/
  • Admission Control • Distributed Throttling • Staged / Pipeline with back pressure • Measure scalability at each stage • Degraded performance • Make progress for admitted requests • At odds with “stateless” / session-less http://www.flickr.com/photos/jayneandd/4450623309/sizes/m/in/photostream/
  • Make Forward Progress • MVCC, vector clocks, & reconciliation • Don’t resurrect objects • always go forward, never go back • "name" is a property of an object, not its unique key • Break the link, garbage collect later • Model “degraded service” performance
  • Request Signing • Stateless - no session tracking to lose or to purge later • X509 - only public information on front- end boxes. More secure against exploit • Shared secret - faster, smaller signature but requires secret info close to request front- end
  • Measure Monitor Respond • Save *everything* *forever* • Histograms / Pareto Chart • tp99.9, tp99, and tp90 • ignore tp50, “average” • http://en.wikipedia.org/wiki/Control_chart • http://www.newrelic.com/ • http://www.splunk.com/ • skewness, kurtosis
  • Control Chart • Day over Day • Same Day,Year over Year • Confidence Intervals “Shewhart stressed that bringing a production process into a state of statistical control, where there is only common-cause variation, and keeping it in control, is necessary to predict future output and to manage a process economically.” • http://en.wikipedia.org/wiki/Control_chart
  • Characteristic Curves
  • Periodicity SLA,Variance, Troubleshooting
  • Data Taxonomy • Precious • Cachable • Expensive • Cheap
  • Consistency • Authoritative vs. Consultative • is_authorized? vs list group
  • Performance • Call length • Cyclomatic Complexity • Request ID flow • Vertical vs Horizontal Scale • tension between unit performance and scalability
  • Failure Domains • EC2 “droplets” • EC2 DNS • Coordinator zones
  • Still with me? Copyright © 2010 Opscode, Inc - All Rights Reserved 39
  • Successes •Sharable “AMI”s •Metadata (Simple and open again) •Open API ( think Eucalyptus) •No API throttling •Primitives •Pay-as you go •Free traffic between S3 and EC2 •Data and Compute together
  • Failures • SOAP makes little girls cry • Amazon Web Services, circa 2006 was > 75% REST or Query • SOAP well supported by commercial vendors, with their libraries • Still *Way* too hard to use. • Commodity business. Driving the bottom out of cost causes quality to suffer. • API vs UI?, User Experience in general • IaaS (Infrastructure as a Service) is insufficient by itself a hangman's noose. EC2, and the other offerings,
  • Where are we going?