Problem                Approach                         Nature of fixes   War Stories                           Performance...
Problem            Approach              Nature of fixes         War StoriesGist          Don’t be out there with a gun loo...
Problem            Approach              Nature of fixes         War StoriesGist          Don’t be out there with a gun loo...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem             Approach               Nature of fixes            War StoriesGist          Don’t be out there with a gu...
Problem            Approach            Nature of fixes          War StoriesContext          App in reference: ThoughtWorks ...
Problem            Approach            Nature of fixes            War StoriesContext          App in reference: ThoughtWork...
Problem            Approach            Nature of fixes            War StoriesContext          App in reference: ThoughtWork...
Problem            Approach            Nature of fixes            War StoriesContext          App in reference: ThoughtWork...
Problem            Approach             Nature of fixes           War StoriesContext          App in reference: ThoughtWork...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem                 Approach                Nature of fixes            War StoriesDon’t go looking for performance issu...
Problem                  Approach                 Nature of fixes             War StoriesRemember...              Don’t be ...
Problem             Approach            Nature of fixes   War StoriesMy app is slow, it doesn’t respond...      In case of ...
Problem             Approach            Nature of fixes   War StoriesMy app is slow, it doesn’t respond...      In case of ...
Problem             Approach             Nature of fixes   War StoriesMy app is slow, it doesn’t respond...      In case of...
Problem             Approach             Nature of fixes   War StoriesMy app is slow, it doesn’t respond...      In case of...
Problem             Approach             Nature of fixes   War StoriesMy app is slow, it doesn’t respond...      In case of...
Problem             Approach             Nature of fixes         War StoriesMy app is slow, it doesn’t respond...      In c...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach   Nature of fixes   War StoriesBenchmark          How slow is slow?
Problem             Approach   Nature of fixes   War StoriesBenchmark          How slow is slow?          How fast is fast?
Problem             Approach            Nature of fixes   War StoriesBenchmark          How slow is slow?          How fast...
Problem             Approach            Nature of fixes   War StoriesBenchmark          How slow is slow?          How fast...
Problem       Approach                 Nature of fixes   War StoriesRemember...                         Do not go overboard...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach                  Nature of fixes   War StoriesTrust in testing                              Wha...
Problem            Approach                  Nature of fixes   War StoriesTrust in testing                              Wha...
Problem            Approach                   Nature of fixes   War StoriesTrust in testing                               W...
Problem            Approach                Nature of fixes   War StoriesTrust in testing                            What if...
Problem       Approach                     Nature of fixes   War StoriesRemember...              Intution tends to work ver...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach        Nature of fixes   War StoriesWhat to measure          Easy and obvious:              Pag...
Problem            Approach            Nature of fixes          War StoriesWhat to measure          Easy and obvious:      ...
Problem       Approach                 Nature of fixes   War StoriesRemember...               Understand your environment. ...
Problem               Approach   Nature of fixes   War StoriesTools to measure          User load              ab          ...
Problem               Approach           Nature of fixes         War StoriesTools to measure          User load            ...
Problem               Approach           Nature of fixes         War StoriesTools to measure          User load            ...
Problem               Approach            Nature of fixes        War StoriesTools to measure          User load            ...
Problem               Approach            Nature of fixes        War StoriesTools to measure          User load            ...
Problem               Approach            Nature of fixes        War StoriesTools to measure          User load            ...
Problem               Approach            Nature of fixes        War StoriesTools to measure          User load            ...
Problem               Approach            Nature of fixes         War StoriesTools to measure          User load           ...
Problem                Approach                Nature of fixes            War StoriesRemember...                          R...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem            Approach              Nature of fixes          War StoriesHow to carry out fixes          Understand libr...
Problem             Approach                   Nature of fixes         War StoriesRemember...                    Requires m...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach            Nature of fixes          War StoriesLock granularity and lockless          When you ...
Problem            Approach            Nature of fixes          War StoriesLock granularity and lockless          When you ...
Problem            Approach              Nature of fixes          War StoriesLock granularity and lockless          When yo...
Problem         Approach                Nature of fixes   War StoriesRemember...              Jumps places, unlike function...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach               Nature of fixes       War StoriesGarbage Collection          When you hear unresp...
Problem            Approach               Nature of fixes       War StoriesGarbage Collection          When you hear unresp...
Problem             Approach                   Nature of fixes               War StoriesGarbage Collection          When yo...
Problem             Approach                   Nature of fixes               War StoriesGarbage Collection          When yo...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach                Nature of fixes            War StoriesSQL Database          Optimize connection ...
Problem            Approach                Nature of fixes            War StoriesSQL Database          Optimize connection ...
Problem            Approach                Nature of fixes            War StoriesSQL Database          Optimize connection ...
Problem              Approach              Nature of fixes            War StoriesSQL Database          Optimize connection ...
Problem            Approach              Nature of fixes   War StoriesSQL Database contd...          Explain and Analyze ar...
Problem            Approach                 Nature of fixes       War StoriesSQL Database contd...          Explain and Ana...
Problem            Approach                 Nature of fixes       War StoriesSQL Database contd...          Explain and Ana...
Problem            Approach                 Nature of fixes       War StoriesSQL Database contd...          Explain and Ana...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach              Nature of fixes            War StoriesContext Switch Thrashing          Too few co...
Problem            Approach              Nature of fixes            War StoriesContext Switch Thrashing          Too few co...
Problem            Approach              Nature of fixes            War StoriesContext Switch Thrashing          Too few co...
Problem            Approach               Nature of fixes            War StoriesContext Switch Thrashing          Too few c...
Problem            Approach               Nature of fixes            War StoriesContext Switch Thrashing          Too few c...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem               Approach             Nature of fixes      War StoriesCommon Pitfalls          Prejudice              ...
Problem               Approach             Nature of fixes            War StoriesCommon Pitfalls          Prejudice        ...
Problem               Approach             Nature of fixes            War StoriesCommon Pitfalls          Prejudice        ...
Problem               Approach             Nature of fixes            War StoriesCommon Pitfalls          Prejudice        ...
Problem               Approach             Nature of fixes              War StoriesCommon Pitfalls          Prejudice      ...
Problem        Approach             Nature of fixes   War StoriesKnow what you are getting into...                Know what...
Problem             Approach               Nature of fixes            War StoriesGist, to reiterate...          Do not go o...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem           Approach       Nature of fixes   War StoriesGarbage churn issues          Debug log statements
Problem           Approach       Nature of fixes   War StoriesGarbage churn issues          Debug log statements          E...
Problem            Approach              Nature of fixes      War StoriesGarbage churn issues          Debug log statements...
Problem            Approach              Nature of fixes      War StoriesGarbage churn issues          Debug log statements...
Problem            Approach              Nature of fixes          War StoriesGarbage churn issues          Debug log statem...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach              Nature of fixes           War StoriesContention and reasoning issues          Lock...
Problem            Approach              Nature of fixes           War StoriesContention and reasoning issues          Lock...
Problem            Approach              Nature of fixes           War StoriesContention and reasoning issues          Lock...
Problem            Approach              Nature of fixes           War StoriesContention and reasoning issues          Lock...
Problem            Approach              Nature of fixes           War StoriesContention and reasoning issues          Lock...
Problem            Approach              Nature of fixes           War StoriesContention and reasoning issues          Lock...
Problem            Approach              Nature of fixes             War StoriesContention and reasoning issues          Lo...
Problem            Approach              Nature of fixes             War StoriesContention and reasoning issues          Lo...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach            Nature of fixes         War StoriesCaching          Eager caching is dangerous(using...
Problem            Approach               Nature of fixes           War StoriesCaching          Eager caching is dangerous(...
Problem            Approach               Nature of fixes           War StoriesCaching          Eager caching is dangerous(...
Problem            Approach               Nature of fixes           War StoriesCaching          Eager caching is dangerous(...
Problem            Approach               Nature of fixes           War StoriesCaching          Eager caching is dangerous(...
Problem            Approach             Nature of fixes         War StoriesCaching contd...          Do not cache uncommite...
Problem            Approach              Nature of fixes           War StoriesCaching contd...          Do not cache uncomm...
Problem            Approach              Nature of fixes           War StoriesCaching contd...          Do not cache uncomm...
Problem            Approach              Nature of fixes           War StoriesCaching contd...          Do not cache uncomm...
Problem              Approach      Nature of fixes   War StoriesOutline      1   Problem             Identify scope        ...
Problem            Approach               Nature of fixes   War StoriesJavascript          JS dom frequent offset/size call...
Problem            Approach               Nature of fixes       War StoriesJavascript          JS dom frequent offset/size ...
Problem            Approach               Nature of fixes         War StoriesJavascript          JS dom frequent offset/siz...
Problem            Approach               Nature of fixes         War StoriesJavascript          JS dom frequent offset/siz...
Problem            Approach               Nature of fixes         War StoriesJavascript          JS dom frequent offset/siz...
Problem            Approach               Nature of fixes         War StoriesJavascript          JS dom frequent offset/siz...
Problem            Approach               Nature of fixes          War StoriesJavascript          JS dom frequent offset/si...
Problem            Approach               Nature of fixes          War StoriesJavascript          JS dom frequent offset/si...
Problem      Approach         Nature of fixes   War StoriesQuestions?                        Questions?
Upcoming SlideShare
Loading in...5
×

Performance Tuning 101

1,498

Published on

The slide deck used for performance tuning basics for an app in managed environment. Uses ThoughtWorks Studios CI/CD server named Go as an example application.

Goes into finding performance problems, pitfalls, anti-patterns, problem patterns for typical performance issues and war-stories extracted from Go performance tuning work.

Performance Tuning 101

  1. 1. Problem Approach Nature of fixes War Stories Performance Tuning 101 An introduction to performance tuning an application in managed environment Pavan KS1 Janmejay Singh2 1 mail: itspanzi@gmail.com blog: http://itspanzi.blogspot.com 2 mail: singh.janmejay@gmail.com blog: http://codehunk.wordpress.com Rootconf Bangalore, May 2012
  2. 2. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems.
  3. 3. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal.
  4. 4. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel.
  5. 5. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure.
  6. 6. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools.
  7. 7. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact.
  8. 8. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first.
  9. 9. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance.
  10. 10. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app).
  11. 11. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted.
  12. 12. Problem Approach Nature of fixes War StoriesGist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted. Don’t jump in the lake without knowing whats waiting inside.
  13. 13. Problem Approach Nature of fixes War StoriesContext App in reference: ThoughtWorks Studios - Go (CI/CD suite)
  14. 14. Problem Approach Nature of fixes War StoriesContext App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp.
  15. 15. Problem Approach Nature of fixes War StoriesContext App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment.
  16. 16. Problem Approach Nature of fixes War StoriesContext App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment. GNU/Linux
  17. 17. Problem Approach Nature of fixes War StoriesContext App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment. GNU/Linux Approach and learning is independent of tech-stack.
  18. 18. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  19. 19. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  20. 20. Problem Approach Nature of fixes War StoriesDon’t go looking for performance issues Doing it proactively is a wild goose chase! Varying parameters and running into a real problem is improbable, if not impossible.
  21. 21. Problem Approach Nature of fixes War StoriesRemember... Don’t be out there with a gun looking for problems. Its easier to reproduce and fix a performance issue when you have seen it happen in a real setup. That way, you also know for sure, it is real.
  22. 22. Problem Approach Nature of fixes War StoriesMy app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow
  23. 23. Problem Approach Nature of fixes War StoriesMy app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow
  24. 24. Problem Approach Nature of fixes War StoriesMy app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule
  25. 25. Problem Approach Nature of fixes War StoriesMy app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late
  26. 26. Problem Approach Nature of fixes War StoriesMy app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late SCM changes reflect after a long time
  27. 27. Problem Approach Nature of fixes War StoriesMy app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late SCM changes reflect after a long time Process is hogging resources(requires 16 cores, 8 gig ram for a basic setup)
  28. 28. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  29. 29. Problem Approach Nature of fixes War StoriesBenchmark How slow is slow?
  30. 30. Problem Approach Nature of fixes War StoriesBenchmark How slow is slow? How fast is fast?
  31. 31. Problem Approach Nature of fixes War StoriesBenchmark How slow is slow? How fast is fast? Benchmark using realistic setup.
  32. 32. Problem Approach Nature of fixes War StoriesBenchmark How slow is slow? How fast is fast? Benchmark using realistic setup. Use a repeatable and automated setup.
  33. 33. Problem Approach Nature of fixes War StoriesRemember... Do not go overboard. Know your end goal.
  34. 34. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  35. 35. Problem Approach Nature of fixes War StoriesTrust in testing What if I start at fix?
  36. 36. Problem Approach Nature of fixes War StoriesTrust in testing What if I start at fix? What if I stop at fix?
  37. 37. Problem Approach Nature of fixes War StoriesTrust in testing What if I start at fix? What if I stop at fix? Why do I need to iterate?
  38. 38. Problem Approach Nature of fixes War StoriesTrust in testing What if I start at fix? What if I stop at fix? Why do I need to iterate? How long should I iterate?
  39. 39. Problem Approach Nature of fixes War StoriesRemember... Intution tends to work very poorly. Belive proof, not gut feel.
  40. 40. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  41. 41. Problem Approach Nature of fixes War StoriesWhat to measure Easy and obvious: Page response time
  42. 42. Problem Approach Nature of fixes War StoriesWhat to measure Easy and obvious: Page response time Not so easy: App throughput Enqueue/Dequeue time for message queues CPU/IO churn Database performance Lock contention Managed environment specific parameters(Memory usage, GC churn etc). Environmental issues(too many processes, too low memory, very slow IO).
  43. 43. Problem Approach Nature of fixes War StoriesRemember... Understand your environment. Know what to measure.
  44. 44. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf
  45. 45. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts
  46. 46. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics.
  47. 47. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify.
  48. 48. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler.
  49. 49. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler.
  50. 50. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler. Managed env specific params: GC logs, GC profiling.
  51. 51. Problem Approach Nature of fixes War StoriesTools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler. Managed env specific params: GC logs, GC profiling. Environmental issues: ps, dmesg, /proc/<pid>/status, /proc/net/dev, syslog, messages.
  52. 52. Problem Approach Nature of fixes War StoriesRemember... Round pegs for round holes. Use appropriate tools. Configure them correctly with least possible overhead.
  53. 53. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  54. 54. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code.
  55. 55. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical.
  56. 56. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem.
  57. 57. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix.
  58. 58. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous.
  59. 59. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads.
  60. 60. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads. Attack one and only one problem.
  61. 61. Problem Approach Nature of fixes War StoriesHow to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads. Attack one and only one problem. Having said all that, correctness always comes first.
  62. 62. Problem Approach Nature of fixes War StoriesRemember... Requires mechanical approach Attack worst problem first Reason every fix Reasoning is of paramount importance in multithreaded app Do one fix at a time Allows easy reasoning and verification, can be reverted Prefer correctness over performance
  63. 63. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  64. 64. Problem Approach Nature of fixes War StoriesLock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse.
  65. 65. Problem Approach Nature of fixes War StoriesLock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse. More than one shared variable mandates synchronization usually, watch out!
  66. 66. Problem Approach Nature of fixes War StoriesLock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse. More than one shared variable mandates synchronization usually, watch out! Atomic operations and globaly unique objects(like interned symbols) are your friends.
  67. 67. Problem Approach Nature of fixes War StoriesRemember... Jumps places, unlike functional bugs Look beyond local impact of a fix
  68. 68. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  69. 69. Problem Approach Nature of fixes War StoriesGarbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library
  70. 70. Problem Approach Nature of fixes War StoriesGarbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs
  71. 71. Problem Approach Nature of fixes War StoriesGarbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs Profile to see if majority of allocations are from just a handful of calls If its several calls with each contributing very little to the pie, don’t even try fixing it.
  72. 72. Problem Approach Nature of fixes War StoriesGarbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs Profile to see if majority of allocations are from just a handful of calls If its several calls with each contributing very little to the pie, don’t even try fixing it. Understand GC ergonomics and each algorithm(if you have alternatives) in detail. Tweek your max/min permitted allocation judiciously, malloc/free are expensive system calls.
  73. 73. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  74. 74. Problem Approach Nature of fixes War StoriesSQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human).
  75. 75. Problem Approach Nature of fixes War StoriesSQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball.
  76. 76. Problem Approach Nature of fixes War StoriesSQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball. Understand your DB settings, MVCC, table-locking scenarios, row-locking scenarios, transaction isolation tradeoffs.
  77. 77. Problem Approach Nature of fixes War StoriesSQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball. Understand your DB settings, MVCC, table-locking scenarios, row-locking scenarios, transaction isolation tradeoffs. Contd...
  78. 78. Problem Approach Nature of fixes War StoriesSQL Database contd... Explain and Analyze are your friends.
  79. 79. Problem Approach Nature of fixes War StoriesSQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity.
  80. 80. Problem Approach Nature of fixes War StoriesSQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity. Every DB may be subtly different(joins can be more expensive than materialzed views(if you have em))
  81. 81. Problem Approach Nature of fixes War StoriesSQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity. Every DB may be subtly different(joins can be more expensive than materialzed views(if you have em)) Denormalization can help crack tough nuts.
  82. 82. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  83. 83. Problem Approach Nature of fixes War StoriesContext Switch Thrashing Too few cores, too many processes will obviously hurt.
  84. 84. Problem Approach Nature of fixes War StoriesContext Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching.
  85. 85. Problem Approach Nature of fixes War StoriesContext Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold.
  86. 86. Problem Approach Nature of fixes War StoriesContext Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold. Too frequent IO will cause io-wait, hence switch. Use of swap can completely kill performance. Swap is for emergency, not for regular usage. Over logging can hurt, be careful! Concept of buffered IO exists for a reason, use it! Use memory mapped IO if it makes sense.
  87. 87. Problem Approach Nature of fixes War StoriesContext Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold. Too frequent IO will cause io-wait, hence switch. Use of swap can completely kill performance. Swap is for emergency, not for regular usage. Over logging can hurt, be careful! Concept of buffered IO exists for a reason, use it! Use memory mapped IO if it makes sense. Assigning very low max memory limit can cause frequent GCs, allocate enough memory so its your functional thread that works and not GC threads.
  88. 88. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  89. 89. Problem Approach Nature of fixes War StoriesCommon Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause
  90. 90. Problem Approach Nature of fixes War StoriesCommon Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit.
  91. 91. Problem Approach Nature of fixes War StoriesCommon Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad.
  92. 92. Problem Approach Nature of fixes War StoriesCommon Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad. Atomic operations and globaly unique objects(like interned symbols) are your friends.
  93. 93. Problem Approach Nature of fixes War StoriesCommon Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad. Atomic operations and globaly unique objects(like interned symbols) are your friends. Unchartred waters ERB garbage churn Strange JVM allocations Single threaded database JRuby class hirarchy modification(runtime level) locks.
  94. 94. Problem Approach Nature of fixes War StoriesKnow what you are getting into... Know what you are getting into...
  95. 95. Problem Approach Nature of fixes War StoriesGist, to reiterate... Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted. Don’t jump in the lake without knowing whats waiting inside.
  96. 96. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  97. 97. Problem Approach Nature of fixes War StoriesGarbage churn issues Debug log statements
  98. 98. Problem Approach Nature of fixes War StoriesGarbage churn issues Debug log statements ERB rendering
  99. 99. Problem Approach Nature of fixes War StoriesGarbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader
  100. 100. Problem Approach Nature of fixes War StoriesGarbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader Stream being dereferenced into a string
  101. 101. Problem Approach Nature of fixes War StoriesGarbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader Stream being dereferenced into a string Thread locals to the rescue(used for caching big reusable objects)
  102. 102. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  103. 103. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking
  104. 104. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods
  105. 105. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock)
  106. 106. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate
  107. 107. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency
  108. 108. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety
  109. 109. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety Take difficult design tradeoffs to keep code requiring a class of lock in one place
  110. 110. Problem Approach Nature of fixes War StoriesContention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety Take difficult design tradeoffs to keep code requiring a class of lock in one place Double check locking can be very helpful in frequently called code
  111. 111. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  112. 112. Problem Approach Nature of fixes War StoriesCaching Eager caching is dangerous(using miss => load => prime => hit works better in practice)
  113. 113. Problem Approach Nature of fixes War StoriesCaching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used
  114. 114. Problem Approach Nature of fixes War StoriesCaching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes
  115. 115. Problem Approach Nature of fixes War StoriesCaching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes Double check cache invalidation flows, its hard, sometimes bleeds across classes.
  116. 116. Problem Approach Nature of fixes War StoriesCaching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes Double check cache invalidation flows, its hard, sometimes bleeds across classes. Be vigilant when using hirarchical caches(no associative-arrays or maps should be cached), its easy to run memory out with this.
  117. 117. Problem Approach Nature of fixes War StoriesCaching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this).
  118. 118. Problem Approach Nature of fixes War StoriesCaching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation.
  119. 119. Problem Approach Nature of fixes War StoriesCaching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation. Understand your caching library, too low overflow limit may cause IO issues.
  120. 120. Problem Approach Nature of fixes War StoriesCaching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation. Understand your caching library, too low overflow limit may cause IO issues. Use one and only one cache instance(its easy to end up in a memory runout with this). We had view fragments, urls, domain objects and view models cached in the same instance.
  121. 121. Problem Approach Nature of fixes War StoriesOutline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  122. 122. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt
  123. 123. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops.
  124. 124. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences.
  125. 125. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible.
  126. 126. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code.
  127. 127. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn.
  128. 128. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn. Use tools like Chromium dev toolset to nail leaks and profile methods.
  129. 129. Problem Approach Nature of fixes War StoriesJavascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn. Use tools like Chromium dev toolset to nail leaks and profile methods. Auto-refreshing dashboards should refresh only visible and changed area.
  130. 130. Problem Approach Nature of fixes War StoriesQuestions? Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×