Download to read offline
Uploaded bySOASTA
Actionable Metrics at Production Scale - LSPE Meetup June 27, 2012
The document outlines various strategies and best practices for managing performance issues in application development, particularly during high traffic periods. It emphasizes the importance of monitoring key metrics in real-time and provides insights on optimizing infrastructure to enhance response times and overall user experience. Key factors such as server resources, database efficiency, and load balancing are highlighted as critical components for maintaining performance under peak conditions.
More Related Content
Similar to Actionable Metrics at Production Scale - LSPE Meetup June 27, 2012
Actionable Metrics at Production Scale - LSPE Meetup June 27, 2012
- 1. 1© 2012 SOASTA.All rights reserved.#lspe Meetup
- 2. 2© 2012 SOASTA.All rights reserved. June 27th , 2012 SOASTA : 12/2009 - Present VP, Product Management Intuit : 4/2007 – 12/2009 Sr. Manger, Engineering TurboTax Online and E-Com ATG : 10/2000 – 4/2007 Sr. Deployment Engineer American Airlines, Best Buy, Target, Turbotax Online, Quicken Online, MySpace, Dennys, Dominos, Mattel, Hallmark, FAA, US Army, AT&T Wireless, Alcatel, Newsweek, Oprah, Neiman Marcus, Plantronics, Kodak, Jcrew, Newell Rubbermaid, Walmart, Target, Paychex, Fidelity Investments
- 3. 3© 2012 SOASTA.All rights reserved. June 27th , 2012 Poor Performance Can be painful.
- 4. 4© 2012 SOASTA.All rights reserved. June 27th , 2012
- 5. 5© 2012 SOASTA.All rights reserved. June 27th , 2012 Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x)Full Scale – To and Above Expected Peak (2x/3x) In ProductionIn ProductionIn ProductionIn Production Usually Live and During Off-HoursUsually Live and During Off-HoursUsually Live and During Off-HoursUsually Live and During Off-Hours Millions of Concurrent UsersMillions of Concurrent Users Tens of Millions of Page Views per HourTens of Millions of Page Views per Hour Many Thousands of Orders per MinuteMany Thousands of Orders per Minute Millions of Concurrent UsersMillions of Concurrent Users Tens of Millions of Page Views per HourTens of Millions of Page Views per Hour Many Thousands of Orders per MinuteMany Thousands of Orders per Minute
- 6. 6© 2012 SOASTA.All rights reserved. June 27th , 2012
- 7. 7© 2012 SOASTA.All rights reserved. June 27th , 2012 ss Memory leaks Scale of Test Stage/Team Dev & Test Release & Deploy Network & Ops CDN file placement Load Balancer configuration Network bandwidth Network configuration DNS routing Inadequate server resources Default configuration settings Unbalanced web serversAuto-scaling failures Latency between systems Slow third-party plug-ins Garbage collection Database thread counts Inefficient database queriesSlow pages Conflict with other apps Test Lab Staging Production (100% +++) Search technology limits Method-level tuning Max sockets exceeded Firewall max capacity Global latency variance Security bottlenecks Tuning for full-scale…and well beyond
- 8. 8© 2012 SOASTA.All rights reserved. June 27th , 2012 Response Time Revenue $$$
- 9. 9© 2012 SOASTA.All rights reserved. June 27th , 2012
- 10. 10© 2012 SOASTA.All rights reserved. June 27th , 2012 Real-Time (within a few seconds)Real-Time (within a few seconds)Real-Time (within a few seconds)Real-Time (within a few seconds) Combine and CorrelateCombine and CorrelateCombine and CorrelateCombine and Correlate
- 11. 11© 2012 SOASTA.All rights reserved. June 27th , 2012 ApplicationApplicationApplicationApplication Level 3 InfrastructureInfrastructureInfrastructureInfrastructure Level 2Level 1 ExternalExternalExternalExternal
- 12. 12© 2012 SOASTA.All rights reserved. June 27th , 2012 ApplicationApplicationApplicationApplication Level 3 InfrastructureInfrastructureInfrastructureInfrastructure Level 2Level 1 ExternalExternalExternalExternal 1.1. Response TimeResponse Time 2.2. Three CriticalThree Critical TransactionsTransactions (counts and(counts and durations)durations) 3.3. ErrorsErrors 4.4. BandwidthBandwidth 1.1. Response TimeResponse Time 2.2. Three CriticalThree Critical TransactionsTransactions (counts and(counts and durations)durations) 3.3. ErrorsErrors 4.4. BandwidthBandwidth 1.1. CPUCPU 2.2. MemoryMemory 3.3. Network and DiskNetwork and Disk I/OI/O 4.4. Load BalancerLoad Balancer 1.1. CPUCPU 2.2. SSL TX/SecSSL TX/Sec 1.1. CPUCPU 2.2. MemoryMemory 3.3. Network and DiskNetwork and Disk I/OI/O 4.4. Load BalancerLoad Balancer 1.1. CPUCPU 2.2. SSL TX/SecSSL TX/Sec 1.1. Container threadContainer thread usageusage 2.2. ContainerContainer memory usagememory usage 3.3. Method responseMethod response timetime 4.4. ConcurrencyConcurrency 1.1. Container threadContainer thread usageusage 2.2. ContainerContainer memory usagememory usage 3.3. Method responseMethod response timetime 4.4. ConcurrencyConcurrency A few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itselfA few at each tier - This can be a ton of data just by itself
- 13. 13© 2012 SOASTA.All rights reserved. June 27th , 2012 Work in LayersWork in Layers • Pick three at eachPick three at each levellevel • Average responseAverage response time is kingtime is king Keep it Real (Time)Keep it Real (Time) Actionable MetricsActionable Metrics • Start from the outside andStart from the outside and work inwardswork inwards • Must be able to easilyMust be able to easily combine and correlatecombine and correlate metrics – it’s aboutmetrics – it’s about reltionshipsreltionships • Data within a fewData within a few seconds isseconds is important to beimportant to be actionableactionable Start SimpleStart Simple
- 14. 14© 2012 SOASTA.All rights reserved. June 27th , 2012 Email: dan@soasta.com Twitter: www.twitter.com/PerfDan LinkedIn: www.linkedin.com/in/danbartow