Performance Monitoring in the Cloud - Gluecon 2011

PerformanceMonitoring inthe Cloud Paul Guth Technical Operations

Agenda Performance and Monitoring and Performance Monitoring How THE CLOUD changes things What you should do (you = cloud developers) What I’d like to seeGluecon - 2011 Cloudscaling - Paul Guth 2

Agenda Performance and Monitoring and Performance Monitoring How THE CLOUD changes things What you should do (you = cloud developers) What I’d like to see We are NOT going to talk about using the cloud to do performance testingGluecon - 2011 Cloudscaling - Paul Guth 2

What Cloudscaling - Paul Guth 3

What is Performance?Gluecon - 2011 Cloudscaling - Paul Guth 4

What is Performance? Numbers speed - rate (184.7 mph) time per unit work (0-60 in 4.1s, 3:04.0min lightning lap) ef ciency (23 mpg) stability (1.00g skidpad) internals (550hp, 510 lb-ft) throughput (4 seats, 13.4 cu ft trunk)Gluecon - 2011 Cloudscaling - Paul Guth 4

What is Performance? Numbers speed - rate (184.7 mph) time per unit work (0-60 in 4.1s, 3:04.0min lightning lap) ef ciency (23 mpg) stability (1.00g skidpad) internals (550hp, 510 lb-ft) throughput (4 seats, 13.4 cu ft trunk) Numbers aren’t everything RWD, live rear axle, 56/44 f/r, airbags, LATCH, ABS - also matterGluecon - 2011 Cloudscaling - Paul Guth 4

What is Monitoring?Gluecon - 2011 Cloudscaling - Paul Guth 5

What is Monitoring? Observing system state through measurements (metrics)Gluecon - 2011 Cloudscaling - Paul Guth 5

What is Monitoring? Observing system state through measurements (metrics) Why? There’s more than one purpose. Detect problems for immediate action Oh noes! Response time just doubled! Provide data for diagnosing problems WTH changed in the last ten minutes? Inform decisions for long-term action What is the current constraint on total throughput? Forecasts for demand are Y by ChristmasGluecon - 2011 Cloudscaling - Paul Guth 5

What Do We Do In Old IT?Gluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include:Gluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include: Activate standby capacityGluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include: Activate standby capacity Turn off featuresGluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include: Activate standby capacity Turn off features Throttle incoming demandGluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include: Activate standby capacity Turn off features Throttle incoming demand “Fix something”Gluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include: Activate standby capacity Turn off features Throttle incoming demand “Fix something” Longer term actions include:Gluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include: Activate standby capacity Turn off features Throttle incoming demand “Fix something” Longer term actions include: Deploying new infrastructure (or removing unneeded)Gluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Immediate actions triggered by performance monitoring include: Activate standby capacity Turn off features Throttle incoming demand “Fix something” Longer term actions include: Deploying new infrastructure (or removing unneeded) “Design something”Gluecon - 2011 Cloudscaling - Paul Guth 6

What Do We Do In Old IT? Common Theme: CapacityGluecon - 2011 Cloudscaling - Paul Guth 6

The Cloud Cloudscaling - Paul Guth 7

Something Cloudy This Way Comes 8Gluecon - 2011 Cloudscaling - Paul Guth

Something Cloudy This Way Comes Traditional IT Adding capacity is: • Expensive • Has a long lead time • Non-reversible • Cheaper when done in large batches • Requires capex outlay 8Gluecon - 2011 Cloudscaling - Paul Guth

Something Cloudy This Way Comes Traditional IT The Cloud Adding capacity is: Adding capacity is: • Expensive • Low marginal cost • Has a long lead time • Quick • Non-reversible • Reversible • Cheaper when done in • Same price in small large batches batches • Requires capex outlay • All opex 8Gluecon - 2011 Cloudscaling - Paul Guth

Something Cloudy This Way Comes Traditional IT The Cloud Adding capacity is: Adding capacity is: • Expensive • Low marginal cost • Has a long lead time • Quick • Non-reversible • Reversible • Cheaper when done in • Same price in small large batches batches • Requires capex outlay • All opex What this means is you can add capacity as an immediate activity, when it used to be a long-term activity. 8Gluecon - 2011 Cloudscaling - Paul Guth

Something Cloudy This Way Comes Traditional IT The Cloud Adding capacity is: Adding capacity is: • Expensive • Low marginal cost • Has a long lead time • Quick • Non-reversible • Reversible • Cheaper when done in • Same price in small large batches batches • Requires capex outlay • All opex What this means is you can add capacity as an immediate activity, when it used to be a long-term activity. In fact, you can automate it. 8Gluecon - 2011 Cloudscaling - Paul Guth

Paradise - The End?Gluecon - 2011 Cloudscaling - Paul Guth 9

Paradise - The End? Wait, all is not perfectGluecon - 2011 Cloudscaling - Paul Guth 9

Paradise - The End? Wait, all is not perfect Sometimes adding capacity is not the right answer Some problems autoscale to in nity Fixing ef ciency may be requiredGluecon - 2011 Cloudscaling - Paul Guth 9

Paradise - The End? Wait, all is not perfect Sometimes adding capacity is not the right answer Some problems autoscale to in nity Fixing ef ciency may be required Adding capacity is cheap but not free Spin up 10k new instances in a day and your controller will want an explanation Transparently balance cost vs bene tGluecon - 2011 Cloudscaling - Paul Guth 9

Paradise - The End? Wait, all is not perfect Sometimes adding capacity is not the right answer Some problems autoscale to in nity Fixing ef ciency may be required Adding capacity is cheap but not free Spin up 10k new instances in a day and your controller will want an explanation Transparently balance cost vs bene t You still need some bufferGluecon - 2011 Cloudscaling - Paul Guth 9

Paradise - The End? Wait, all is not perfect Sometimes adding capacity is not the right answer Some problems autoscale to in nity Fixing ef ciency may be required Adding capacity is cheap but not free Spin up 10k new instances in a day and your controller will want an explanation Transparently balance cost vs bene t You still need some buffer Your automation needs limits Just Say No to SkynetGluecon - 2011 Cloudscaling - Paul Guth 9

Paradise - The End? Wait, all is not perfect Sometimes adding capacity is not the right answer Some problems autoscale to in nity Fixing ef ciency may be required Adding capacity is cheap but not free Spin up 10k new instances in a day and your controller will want an explanation Transparently balance cost vs bene t You still need some buffer Your automation needs limits Just Say No to Skynet “Auto-scale me if you want to live!”Gluecon - 2011 Cloudscaling - Paul Guth 9

What’s a CloudDeveloper to Do? Cloudscaling - Paul Guth 10

What Do You Do?Gluecon - 2011 Cloudscaling - Paul Guth 11

What Do You Do? What to measure? iops, cpu util, memfree, queue latency? USELESS! (*)Gluecon - 2011 Cloudscaling - Paul Guth 11

What Do You Do? What to measure? iops, cpu util, memfree, queue latency? USELESS! (*) First measure what your customers care about What do they pay you for? Response time, load time, functional correctnessGluecon - 2011 Cloudscaling - Paul Guth 11

What Do You Do? What to measure? iops, cpu util, memfree, queue latency? USELESS! (*) First measure what your customers care about What do they pay you for? Response time, load time, functional correctness Monitor services (from customer perspective), not serversGluecon - 2011 Cloudscaling - Paul Guth 11

What Do You Do? What to measure? iops, cpu util, memfree, queue latency? USELESS! (*) First measure what your customers care about What do they pay you for? Response time, load time, functional correctness Monitor services (from customer perspective), not servers Monitor your cost as well (COGS) - costs more variable nowGluecon - 2011 Cloudscaling - Paul Guth 11

What Do You Do? What to measure? iops, cpu util, memfree, queue latency? USELESS! (*) First measure what your customers care about What do they pay you for? Response time, load time, functional correctness Monitor services (from customer perspective), not servers Monitor your cost as well (COGS) - costs more variable now (*) NOTE: Not actually uselessGluecon - 2011 Cloudscaling - Paul Guth 11

Thought ExperimentGluecon - 2011 Cloudscaling - Paul Guth 12

Thought Experiment 0300 SundayGluecon - 2011 Cloudscaling - Paul Guth 12

Thought Experiment 0300 Sunday CPU Utilization on your cluster increases to 100%Gluecon - 2011 Cloudscaling - Paul Guth 12

Thought Experiment 0300 Sunday CPU Utilization on your cluster increases to 100% External service monitoring shows no problemsGluecon - 2011 Cloudscaling - Paul Guth 12

Thought Experiment 0300 Sunday CPU Utilization on your cluster increases to 100% External service monitoring shows no problems What do yo do?Gluecon - 2011 Cloudscaling - Paul Guth 12

Thought Experiment 0300 Sunday CPU Utilization on your cluster increases to 100% External service monitoring shows no problems What do yo do? Go back to sleep!Gluecon - 2011 Cloudscaling - Paul Guth 12

Thought Experiment 0300 Sunday CPU Utilization on your cluster increases to 100% External service monitoring shows no problems What do yo do? Go back to sleep! Please investigate on Monday, you’re probably wasting moneyGluecon - 2011 Cloudscaling - Paul Guth 12

Monitoring Hierarchy Customer Services Application Metrics Server MetricsGluecon - 2011 Cloudscaling - Paul Guth 13

Other Things to DoGluecon - 2011 Cloudscaling - Paul Guth 14

Other Things to Do Record religiously all data around resource callsGluecon - 2011 Cloudscaling - Paul Guth 14

Other Things to Do Record religiously all data around resource calls Put in monitoring/metrics from the start!Gluecon - 2011 Cloudscaling - Paul Guth 14

Other Things to Do Record religiously all data around resource calls Put in monitoring/metrics from the start! Make it trivial (for devs) to record metrics, and incentivize them collectd, graphite, etc Build it into your framework/platform of choiceGluecon - 2011 Cloudscaling - Paul Guth 14

Other Things to Do Record religiously all data around resource calls Put in monitoring/metrics from the start! Make it trivial (for devs) to record metrics, and incentivize them collectd, graphite, etc Build it into your framework/platform of choice Make sure this monitoring scales out automatically when new instances appear and is retained when instances disappear As much of this monitoring as possible should be at the cluster, not instance levelGluecon - 2011 Cloudscaling - Paul Guth 14

Other Things to Do Record religiously all data around resource calls Put in monitoring/metrics from the start! Make it trivial (for devs) to record metrics, and incentivize them collectd, graphite, etc Build it into your framework/platform of choice Make sure this monitoring scales out automatically when new instances appear and is retained when instances disappear As much of this monitoring as possible should be at the cluster, not instance level Use much more care when guring out what to alert about - start with just customer services False positives can be killers Use data for diagnosis rst, then learn when to alertGluecon - 2011 Cloudscaling - Paul Guth 14

Other Other Things to DoGluecon - 2011 Cloudscaling - Paul Guth 15

Other Other Things to Do Treat dependencies as if they’re vitally important! Have model. Have API. Use model API Have tools to visualize the model Leverage it in your other toolsGluecon - 2011 Cloudscaling - Paul Guth 15

Other Other Things to Do Treat dependencies as if they’re vitally important! Have model. Have API. Use model API Have tools to visualize the model Leverage it in your other tools Automate handling all anticipated conditionsGluecon - 2011 Cloudscaling - Paul Guth 15

Other Other Things to Do Treat dependencies as if they’re vitally important! Have model. Have API. Use model API Have tools to visualize the model Leverage it in your other tools Automate handling all anticipated conditions Learn learn learnGluecon - 2011 Cloudscaling - Paul Guth 15

Other Other Things to Do Treat dependencies as if they’re vitally important! Have model. Have API. Use model API Have tools to visualize the model Leverage it in your other tools Automate handling all anticipated conditions Learn learn learn Measure customer experience!Gluecon - 2011 Cloudscaling - Paul Guth 15

Other Other Things to Do Treat dependencies as if they’re vitally important! Have model. Have API. Use model API Have tools to visualize the model Leverage it in your other tools Automate handling all anticipated conditions Learn learn learn Measure customer experience! Test in production (learn from production at least)Gluecon - 2011 Cloudscaling - Paul Guth 15

What The World Needs Now Cloudscaling - Paul Guth 16

The Ideal Monitoring DashboardGluecon - 2011 Cloudscaling - Paul Guth 17

The Ideal Monitoring Dashboard OK $/sec (in): 1,000 $/sec (out): 1,000Gluecon - 2011 Cloudscaling - Paul Guth 17

DashboardsGluecon - 2011 Cloudscaling - Paul Guth 18

Dashboards Actionable data! Long lists of stuff that’s OK are useless “Event consoles” are useless - I care about current state, not what happened ve minutes ago Too much data is worse than no dataGluecon - 2011 Cloudscaling - Paul Guth 18

Dashboards Actionable data! Long lists of stuff that’s OK are useless “Event consoles” are useless - I care about current state, not what happened ve minutes ago Too much data is worse than no data At top-level, show only customer servicesGluecon - 2011 Cloudscaling - Paul Guth 18

Dashboards Actionable data! Long lists of stuff that’s OK are useless “Event consoles” are useless - I care about current state, not what happened ve minutes ago Too much data is worse than no data At top-level, show only customer services Drill-down to what you wantGluecon - 2011 Cloudscaling - Paul Guth 18

Dashboards Actionable data! Long lists of stuff that’s OK are useless “Event consoles” are useless - I care about current state, not what happened ve minutes ago Too much data is worse than no data At top-level, show only customer services Drill-down to what you want Filter easily to narrow inGluecon - 2011 Cloudscaling - Paul Guth 18

Dashboards Actionable data! Long lists of stuff that’s OK are useless “Event consoles” are useless - I care about current state, not what happened ve minutes ago Too much data is worse than no data At top-level, show only customer services Drill-down to what you want Filter easily to narrow in Save the trees, have a search boxGluecon - 2011 Cloudscaling - Paul Guth 18

Dashboards Actionable data! Long lists of stuff that’s OK are useless “Event consoles” are useless - I care about current state, not what happened ve minutes ago Too much data is worse than no data At top-level, show only customer services Drill-down to what you want Filter easily to narrow in Save the trees, have a search box Add arbitrary time-series data to any chart - including changelogs, business metricsGluecon - 2011 Cloudscaling - Paul Guth 18

One Size Fits One Use different interfaces for different purposes and/or different audiences vsGluecon - 2011 Cloudscaling - Paul Guth 19

Summary Performance Monitoring and Capacity Management are joined at the hip The Cloud enables automated, immediate capacity xes The price of automation is eternal vigilance Monitor your customer-facing services rst Make it so easy to collect metrics that you’ll have tons and tons of them Magic dashboard make Paul happy!Gluecon - 2011 Cloudscaling - Paul Guth 20

Thank You! g < at > cloudscaling d0t com @pguthebGluecon - 2011 Cloudscaling - Paul Guth 21

Performance Monitoring in the Cloud - Gluecon 2011

by Paul Guth on May 25, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Performance Monitoring in the Cloud - Gluecon 2011 — Presentation Transcript