Your SlideShare is downloading. ×
0
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Intro to linux performance analysis
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Intro to linux performance analysis

315

Published on

LOPSA SD 2014.03.27 Presentation on Linux Performance Analysis …

LOPSA SD 2014.03.27 Presentation on Linux Performance Analysis

An introduction using the USE method and showing how several tools fit into those resource evaluations.

Published in: Internet
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
315
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
27
Comments
0
Likes
6
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. Intro to Linux Performance Analysis Chris McEniry LOPSA-SD March 27, 2014
  2. Me • Systems Architect • Sony Network Entertainment • 18 years running stuff • Majority of the last 14 years: medium-large Internet services
  3. Read this book… And look here: http://www.brendangregg.com/ http://www.brendangregg.com/ methodology.html http://www.brendangregg.com/Slides/ LISA2012_methodologies.pdf http://www.amazon.com/Systems-Performance-Enterprise-Brendan-Gregg/dp/0133390098
  4. The website is down!!! It’s just too slow! The DB is too slow! The disk is too slow! SLOW!!! http://farm4.staticflickr.com/3190/2976755407_6a6a574596_o.jpg
  5. SLOW!!!! • What does slow mean anyways? • Is it not transferring fast enough? • Is it handling (not) too many requests? http://commons.wikimedia.org/wiki/File:United_States_sign_-_Slow_Traffic_Ahead.svg
  6. Slow can mean… • Latency: How long it takes • ms, s, request time, etc • Throughput: How much can happen at the same time • bandwidth, IOPS, rps, tps, etc http://upload.wikimedia.org/wikipedia/commons/2/2e/Miniature_DNF_Dictionary_055_ubt.JPG
  7. Slowness comes from… • Full utilization of a resource • Waiting in a saturated queue • Generated errors! ! • The USE Method http://farm6.staticflickr.com/5181/5614813544_a30d693a50_o.jpg
  8. Utilization • You have fully used up what’s been allocated • aka 5 lb bag http://farm3.staticflickr.com/2524/4000641774_3331fe06fb_o.jpg
  9. Saturation • Waiting for someone else to get done so you can do yours • Typically because a resource is fully utilized, but not necessarily directly http://www.fotocommunity.com/pc/pc/display/30396619
  10. Errors • Dropped packets • Incorrect responses • Deadlocks • Timeouts ! • Not all failures fail fast http://farm8.staticflickr.com/7001/6509400855_aaaf915871_b.jpg
  11. How do we determine? • Different types of tools for different examinations • Depends on what you’re looking for (which can be a problem in and of itself) http://farm5.staticflickr.com/4083/5086955738_61f6455ace_b.jpg
  12. Resource vs Transaction • Do you care if… • a CPU is maxed out? • processes are blocked? • packets are lost? • or if… • a user’s request fails? • a user gives up on waiting for a response?
  13. Maturity • Tracing tools, especially using in production, requires a level of maturity • I’m not that mature… ;) • No, really just focusing on the basics first http://upload.wikimedia.org/wikipedia/commons/b/bd/OFLC_large_R18%2B.svg
  14. http://image.slidesharecdn.com/scalelinuxperformance-130224171331-phpapp01/95/slide-15-638.jpg?cb=1362166290
  15. http://image.slidesharecdn.com/scalelinuxperformance-130224171331-phpapp01/95/slide-16-638.jpg?cb=1362166290
  16. General
  17. ? /var/log/messages
  18. Errors ! (mostly - sometimes stats go here) /var/log/messages
  19. CPU
  20. ? uptime
  21. Saturation of the scheduler uptime
  22. ? top
  23. top Saturation Utilization
  24. Memory
  25. ? free
  26. Utilization free
  27. ? vmstat
  28. vmstat Saturation Utilization Counts
  29. ? slabtop
  30. Utilization slabtop
  31. Disk
  32. ? df
  33. Utilization df
  34. ? iostat -x
  35. Maybe you can get additional utilization if you know the max r/s or w/s - but not as clear based on different properties. iostat -x
  36. IO (Network)
  37. ? ping
  38. Errors ping
  39. ? netstat
  40. Saturation netstat
  41. ? netstat -s
  42. Errors netstat -s
  43. ? ifconfig
  44. ifconfig Saturation Utilization Errors
  45. What are your examples? http://upload.wikimedia.org/wikipedia/commons/f/f3/Uncle_Sam_(pointing_finger).jpg
  46. Applications
  47. Running out of Apache Threads • Lots of incoming requests • Apache hits ServerLimit of threads (Utilization!) • Requests start to get stuck in TCP backlog (Saturation!) • Apache endpoints are removed from load balancers (Error!) • Fail! http://upload.wikimedia.org/wikipedia/commons/9/96/Colorful_Threads_(3965274345).jpg
  48. Cold DB Start • DB’s like to be in memory, but can’t start that way • All data requests go to disk (which is SAN backed) • SAN controller CPU gets maxed out (Utilization!) • HBA queues get deep (Saturation!) • Requests timeout (Error!) • Fail!
  49. Summary
  50. Methods > Tools • Don’t let tools get in the way of solutions • It’s easy to think that all your missing a tool. • But are you actually following a method to your performance madness? http://upload.wikimedia.org/wikipedia/commons/6/6d/Three_Card_Monte.jpg
  51. Anti-Methods • Blame Someone Else • Streetlight • Drunk Man • Random Change • Passive Benchmark ! • Don’t do these… http://www.brendangregg.com/methodology.html http://upload.wikimedia.org/wikipedia/commons/a/af/Villainc.svg
  52. Methods • Ad Hoc Checklist • Problem Statement • Scientific • Workload Characterization • Drill-down Analysis • By-layer • Latency Analysis • Tools • Stack Profile • Off-CPU Analysis • Thread State Analysis • Active Benchmark http://www.brendangregg.com/methodology.html http://memegenerator.net/instance/9192015
  53. Linux Performance Tools Chris McEniry LOPSA-SD March 27, 2014

×