1© 2012 SOASTA. All rights reserved.#lspe Meetup
2© 2012 SOASTA. All rights reserved. June 27th
, 2012
SOASTA : 12/2009 - Present
VP, Product Management
Intuit : 4/2007 – ...
3© 2012 SOASTA. All rights reserved. June 27th
, 2012
Poor
Performance
Can be painful.
4© 2012 SOASTA. All rights reserved. June 27th
, 2012
5© 2012 SOASTA. All rights reserved. June 27th
, 2012
Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and A...
6© 2012 SOASTA. All rights reserved. June 27th
, 2012
7© 2012 SOASTA. All rights reserved. June 27th
, 2012
ss
Memory
leaks
Scale of Test
Stage/Team
Dev
& Test
Release
& Deploy...
8© 2012 SOASTA. All rights reserved. June 27th
, 2012
Response Time
Revenue
$$$



9© 2012 SOASTA. All rights reserved. June 27th
, 2012
10© 2012 SOASTA. All rights reserved. June 27th
, 2012
Real-Time (within a few seconds)Real-Time (within a few seconds)Rea...
11© 2012 SOASTA. All rights reserved. June 27th
, 2012
ApplicationApplicationApplicationApplication
Level 3
Infrastructure...
12© 2012 SOASTA. All rights reserved. June 27th
, 2012
ApplicationApplicationApplicationApplication
Level 3
Infrastructure...
13© 2012 SOASTA. All rights reserved. June 27th
, 2012
Work in LayersWork in Layers
• Pick three at eachPick three at each...
14© 2012 SOASTA. All rights reserved. June 27th
, 2012
Email: dan@soasta.com
Twitter: www.twitter.com/PerfDan
LinkedIn: ww...
Upcoming SlideShare
Loading in …5
×

Actionable Metrics at Production Scale - LSPE Meetup June 27, 2012

95 views

Published on

Presentation to Large Scale Production Engineering meetup at Yahoo campus on June 27, 2012. Dan Bartow, VP Product Management at SOASTA discussed critical performance metrics.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Actionable Metrics at Production Scale - LSPE Meetup June 27, 2012

  1. 1. 1© 2012 SOASTA. All rights reserved.#lspe Meetup
  2. 2. 2© 2012 SOASTA. All rights reserved. June 27th , 2012 SOASTA : 12/2009 - Present VP, Product Management Intuit : 4/2007 – 12/2009 Sr. Manger, Engineering TurboTax Online and E-Com ATG : 10/2000 – 4/2007 Sr. Deployment Engineer American Airlines, Best Buy, Target, Turbotax Online, Quicken Online, MySpace, Dennys, Dominos, Mattel, Hallmark, FAA, US Army, AT&T Wireless, Alcatel, Newsweek, Oprah, Neiman Marcus, Plantronics, Kodak, Jcrew, Newell Rubbermaid, Walmart, Target, Paychex, Fidelity Investments
  3. 3. 3© 2012 SOASTA. All rights reserved. June 27th , 2012 Poor Performance Can be painful.
  4. 4. 4© 2012 SOASTA. All rights reserved. June 27th , 2012
  5. 5. 5© 2012 SOASTA. All rights reserved. June 27th , 2012 Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x) In ProductionIn ProductionIn ProductionIn Production Usually Live and During Off-HoursUsually Live and During Off-HoursUsually Live and During Off-HoursUsually Live and During Off-Hours Millions of Concurrent UsersMillions of Concurrent Users Tens of Millions of Page Views per HourTens of Millions of Page Views per Hour Many Thousands of Orders per MinuteMany Thousands of Orders per Minute Millions of Concurrent UsersMillions of Concurrent Users Tens of Millions of Page Views per HourTens of Millions of Page Views per Hour Many Thousands of Orders per MinuteMany Thousands of Orders per Minute
  6. 6. 6© 2012 SOASTA. All rights reserved. June 27th , 2012
  7. 7. 7© 2012 SOASTA. All rights reserved. June 27th , 2012 ss Memory leaks Scale of Test Stage/Team Dev & Test Release & Deploy Network & Ops CDN file placement Load Balancer configuration Network bandwidth Network configuration DNS routing Inadequate server resources Default configuration settings Unbalanced web serversAuto-scaling failures Latency between systems Slow third-party plug-ins Garbage collection Database thread counts Inefficient database queriesSlow pages Conflict with other apps Test Lab Staging Production (100% +++) Search technology limits Method-level tuning Max sockets exceeded Firewall max capacity Global latency variance Security bottlenecks Tuning for full-scale…and well beyond
  8. 8. 8© 2012 SOASTA. All rights reserved. June 27th , 2012 Response Time Revenue $$$   
  9. 9. 9© 2012 SOASTA. All rights reserved. June 27th , 2012
  10. 10. 10© 2012 SOASTA. All rights reserved. June 27th , 2012 Real-Time (within a few seconds)Real-Time (within a few seconds)Real-Time (within a few seconds)Real-Time (within a few seconds) Combine and CorrelateCombine and CorrelateCombine and CorrelateCombine and Correlate
  11. 11. 11© 2012 SOASTA. All rights reserved. June 27th , 2012 ApplicationApplicationApplicationApplication Level 3 InfrastructureInfrastructureInfrastructureInfrastructure Level 2Level 1 ExternalExternalExternalExternal
  12. 12. 12© 2012 SOASTA. All rights reserved. June 27th , 2012 ApplicationApplicationApplicationApplication Level 3 InfrastructureInfrastructureInfrastructureInfrastructure Level 2Level 1 ExternalExternalExternalExternal 1.1. Response TimeResponse Time 2.2. Three CriticalThree Critical TransactionsTransactions (counts and(counts and durations)durations) 3.3. ErrorsErrors 4.4. BandwidthBandwidth 1.1. Response TimeResponse Time 2.2. Three CriticalThree Critical TransactionsTransactions (counts and(counts and durations)durations) 3.3. ErrorsErrors 4.4. BandwidthBandwidth 1.1. CPUCPU 2.2. MemoryMemory 3.3. Network and DiskNetwork and Disk I/OI/O 4.4. Load BalancerLoad Balancer 1.1. CPUCPU 2.2. SSL TX/SecSSL TX/Sec 1.1. CPUCPU 2.2. MemoryMemory 3.3. Network and DiskNetwork and Disk I/OI/O 4.4. Load BalancerLoad Balancer 1.1. CPUCPU 2.2. SSL TX/SecSSL TX/Sec 1.1. Container threadContainer thread usageusage 2.2. ContainerContainer memory usagememory usage 3.3. Method responseMethod response timetime 4.4. ConcurrencyConcurrency 1.1. Container threadContainer thread usageusage 2.2. ContainerContainer memory usagememory usage 3.3. Method responseMethod response timetime 4.4. ConcurrencyConcurrency A few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itself
  13. 13. 13© 2012 SOASTA. All rights reserved. June 27th , 2012 Work in LayersWork in Layers • Pick three at eachPick three at each levellevel • Average responseAverage response time is kingtime is king Keep it Real (Time)Keep it Real (Time) Actionable MetricsActionable Metrics • Start from the outside andStart from the outside and work inwardswork inwards • Must be able to easilyMust be able to easily combine and correlatecombine and correlate metrics – it’s aboutmetrics – it’s about reltionshipsreltionships • Data within a fewData within a few seconds isseconds is important to beimportant to be actionableactionable Start SimpleStart Simple
  14. 14. 14© 2012 SOASTA. All rights reserved. June 27th , 2012 Email: dan@soasta.com Twitter: www.twitter.com/PerfDan LinkedIn: www.linkedin.com/in/danbartow

×