SlideShare a Scribd company logo
1 of 130
Download to read offline
Problem                Approach                         Nature of fixes   War Stories




                           Performance Tuning 101
          An introduction to performance tuning an application in managed
                                    environment


                          Pavan KS1             Janmejay Singh2

                                     1 mail: itspanzi@gmail.com

                                  blog: http://itspanzi.blogspot.com
                              2 mail: singh.janmejay@gmail.com

                             blog: http://codehunk.wordpress.com


                          Rootconf Bangalore, May 2012
Problem            Approach              Nature of fixes         War Stories



Gist
          Don’t be out there with a gun looking for problems.
Problem            Approach              Nature of fixes         War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
          Jumps places, unlike functional bugs, look beyond local
          impact.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
          Jumps places, unlike functional bugs, look beyond local
          impact.
          Requires mechanical approach, attack worst problem first.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
          Jumps places, unlike functional bugs, look beyond local
          impact.
          Requires mechanical approach, attack worst problem first.
          Prefer correctness over performance.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
          Jumps places, unlike functional bugs, look beyond local
          impact.
          Requires mechanical approach, attack worst problem first.
          Prefer correctness over performance.
          Reasoning every fix (of paramount importance in
          multithreaded app).
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
          Jumps places, unlike functional bugs, look beyond local
          impact.
          Requires mechanical approach, attack worst problem first.
          Prefer correctness over performance.
          Reasoning every fix (of paramount importance in
          multithreaded app).
          Do one fix at a time, allows easy reasoning and
          verification, can be reverted.
Problem             Approach               Nature of fixes            War Stories



Gist
          Don’t be out there with a gun looking for problems.
          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
          Jumps places, unlike functional bugs, look beyond local
          impact.
          Requires mechanical approach, attack worst problem first.
          Prefer correctness over performance.
          Reasoning every fix (of paramount importance in
          multithreaded app).
          Do one fix at a time, allows easy reasoning and
          verification, can be reverted.
          Don’t jump in the lake without knowing whats waiting
          inside.
Problem            Approach            Nature of fixes          War Stories



Context




          App in reference: ThoughtWorks Studios - Go (CI/CD
          suite)
Problem            Approach            Nature of fixes            War Stories



Context




          App in reference: ThoughtWorks Studios - Go (CI/CD
          suite)
          Java (backend) and JRuby on Rails (frontend) webapp.
Problem            Approach            Nature of fixes            War Stories



Context




          App in reference: ThoughtWorks Studios - Go (CI/CD
          suite)
          Java (backend) and JRuby on Rails (frontend) webapp.
          Managed environment.
Problem            Approach            Nature of fixes            War Stories



Context




          App in reference: ThoughtWorks Studios - Go (CI/CD
          suite)
          Java (backend) and JRuby on Rails (frontend) webapp.
          Managed environment.
          GNU/Linux
Problem            Approach             Nature of fixes           War Stories



Context




          App in reference: ThoughtWorks Studios - Go (CI/CD
          suite)
          Java (backend) and JRuby on Rails (frontend) webapp.
          Managed environment.
          GNU/Linux
          Approach and learning is independent of tech-stack.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem                 Approach                Nature of fixes            War Stories



Don’t go looking for performance issues




                  Doing it proactively is a wild goose chase!

          Varying parameters and running into a real problem is improbable, if
                                   not impossible.
Problem                  Approach                 Nature of fixes             War Stories



Remember...




              Don’t be out there with a gun looking for problems.

           Its easier to reproduce and fix a performance issue when you have
          seen it happen in a real setup. That way, you also know for sure, it is
                                           real.
Problem             Approach            Nature of fixes   War Stories



My app is slow, it doesn’t respond...



      In case of Go, this could mean,
          The dashboard is slow
Problem             Approach            Nature of fixes   War Stories



My app is slow, it doesn’t respond...



      In case of Go, this could mean,
          The dashboard is slow
          Javascript is slow
Problem             Approach             Nature of fixes   War Stories



My app is slow, it doesn’t respond...



      In case of Go, this could mean,
          The dashboard is slow
          Javascript is slow
          Builds taking too long to schedule
Problem             Approach             Nature of fixes   War Stories



My app is slow, it doesn’t respond...



      In case of Go, this could mean,
          The dashboard is slow
          Javascript is slow
          Builds taking too long to schedule
          Worker processes picking up work too late
Problem             Approach             Nature of fixes   War Stories



My app is slow, it doesn’t respond...



      In case of Go, this could mean,
          The dashboard is slow
          Javascript is slow
          Builds taking too long to schedule
          Worker processes picking up work too late
          SCM changes reflect after a long time
Problem             Approach             Nature of fixes         War Stories



My app is slow, it doesn’t respond...



      In case of Go, this could mean,
          The dashboard is slow
          Javascript is slow
          Builds taking too long to schedule
          Worker processes picking up work too late
          SCM changes reflect after a long time
          Process is hogging resources(requires 16 cores, 8 gig ram
          for a basic setup)
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach   Nature of fixes   War Stories



Benchmark




          How slow is slow?
Problem             Approach   Nature of fixes   War Stories



Benchmark




          How slow is slow?
          How fast is fast?
Problem             Approach            Nature of fixes   War Stories



Benchmark




          How slow is slow?
          How fast is fast?
          Benchmark using realistic setup.
Problem             Approach            Nature of fixes   War Stories



Benchmark




          How slow is slow?
          How fast is fast?
          Benchmark using realistic setup.
          Use a repeatable and automated setup.
Problem       Approach                 Nature of fixes   War Stories



Remember...




                         Do not go overboard.

                          Know your end goal.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach                  Nature of fixes   War Stories



Trust in testing




                              What if I start at fix?
Problem            Approach                  Nature of fixes   War Stories



Trust in testing




                              What if I start at fix?
                              What if I stop at fix?
Problem            Approach                   Nature of fixes   War Stories



Trust in testing




                               What if I start at fix?
                               What if I stop at fix?
                              Why do I need to iterate?
Problem            Approach                Nature of fixes   War Stories



Trust in testing




                            What if I start at fix?
                            What if I stop at fix?
                          Why do I need to iterate?
                          How long should I iterate?
Problem       Approach                     Nature of fixes   War Stories



Remember...




              Intution tends to work very poorly.

                         Belive proof, not gut feel.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach        Nature of fixes   War Stories



What to measure


          Easy and obvious:
              Page response time
Problem            Approach            Nature of fixes          War Stories



What to measure


          Easy and obvious:
              Page response time
          Not so easy:
              App throughput
              Enqueue/Dequeue time for message queues
              CPU/IO churn
              Database performance
              Lock contention
              Managed environment specific parameters(Memory usage,
              GC churn etc).
              Environmental issues(too many processes, too low
              memory, very slow IO).
Problem       Approach                 Nature of fixes   War Stories



Remember...




               Understand your environment.

                         Know what to measure.
Problem               Approach   Nature of fixes   War Stories



Tools to measure

          User load
              ab
              htperf
Problem               Approach           Nature of fixes         War Stories



Tools to measure

          User load
              ab
              htperf
          App throughput: measure app-level
          churn(creation/processing/deletion/delivery) counts
Problem               Approach           Nature of fixes         War Stories



Tools to measure

          User load
              ab
              htperf
          App throughput: measure app-level
          churn(creation/processing/deletion/delivery) counts
          Enqueue/Dequeue time: Use broker’s management
          console/logs/statistics.
Problem               Approach            Nature of fixes        War Stories



Tools to measure

          User load
              ab
              htperf
          App throughput: measure app-level
          churn(creation/processing/deletion/delivery) counts
          Enqueue/Dequeue time: Use broker’s management
          console/logs/statistics.
          CPU/IO churn: Load Avg, iostat, klogd, inotify.
Problem               Approach            Nature of fixes        War Stories



Tools to measure

          User load
              ab
              htperf
          App throughput: measure app-level
          churn(creation/processing/deletion/delivery) counts
          Enqueue/Dequeue time: Use broker’s management
          console/logs/statistics.
          CPU/IO churn: Load Avg, iostat, klogd, inotify.
          Database performance: Profiler.
Problem               Approach            Nature of fixes        War Stories



Tools to measure

          User load
              ab
              htperf
          App throughput: measure app-level
          churn(creation/processing/deletion/delivery) counts
          Enqueue/Dequeue time: Use broker’s management
          console/logs/statistics.
          CPU/IO churn: Load Avg, iostat, klogd, inotify.
          Database performance: Profiler.
          Lock contention: Profiler.
Problem               Approach            Nature of fixes        War Stories



Tools to measure

          User load
              ab
              htperf
          App throughput: measure app-level
          churn(creation/processing/deletion/delivery) counts
          Enqueue/Dequeue time: Use broker’s management
          console/logs/statistics.
          CPU/IO churn: Load Avg, iostat, klogd, inotify.
          Database performance: Profiler.
          Lock contention: Profiler.
          Managed env specific params: GC logs, GC profiling.
Problem               Approach            Nature of fixes         War Stories



Tools to measure

          User load
              ab
              htperf
          App throughput: measure app-level
          churn(creation/processing/deletion/delivery) counts
          Enqueue/Dequeue time: Use broker’s management
          console/logs/statistics.
          CPU/IO churn: Load Avg, iostat, klogd, inotify.
          Database performance: Profiler.
          Lock contention: Profiler.
          Managed env specific params: GC logs, GC profiling.
          Environmental issues: ps, dmesg, /proc/<pid>/status,
          /proc/net/dev, syslog, messages.
Problem                Approach                Nature of fixes            War Stories



Remember...




                          Round pegs for round holes.

          Use appropriate tools. Configure them correctly with least possible
                                     overhead.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
          Be mechanical.
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
          Be mechanical.
          Use exhaggerated load to maginify the problem.
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
          Be mechanical.
          Use exhaggerated load to maginify the problem.
          Use the same load profile to test your fix.
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
          Be mechanical.
          Use exhaggerated load to maginify the problem.
          Use the same load profile to test your fix.
          Be rigorous.
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
          Be mechanical.
          Use exhaggerated load to maginify the problem.
          Use the same load profile to test your fix.
          Be rigorous.
          Mob pair on complicated fixes, especially fixes involving
          data mutation across threads.
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
          Be mechanical.
          Use exhaggerated load to maginify the problem.
          Use the same load profile to test your fix.
          Be rigorous.
          Mob pair on complicated fixes, especially fixes involving
          data mutation across threads.
          Attack one and only one problem.
Problem            Approach              Nature of fixes          War Stories



How to carry out fixes


          Understand libraries: read fine print in documentation, or
          better yet, read code.
          Be mechanical.
          Use exhaggerated load to maginify the problem.
          Use the same load profile to test your fix.
          Be rigorous.
          Mob pair on complicated fixes, especially fixes involving
          data mutation across threads.
          Attack one and only one problem.
          Having said all that, correctness always comes first.
Problem             Approach                   Nature of fixes         War Stories



Remember...


                    Requires mechanical approach

                               Attack worst problem first

                                 Reason every fix

          Reasoning is of paramount importance in multithreaded app

                               Do one fix at a time

            Allows easy reasoning and verification, can be reverted

                 Prefer correctness over performance
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach            Nature of fixes          War Stories



Lock granularity and lockless




          When you fix lock contention somewhere, ensure it hasn’t
          moved elsewhere, making overall performance worse.
Problem            Approach            Nature of fixes          War Stories



Lock granularity and lockless




          When you fix lock contention somewhere, ensure it hasn’t
          moved elsewhere, making overall performance worse.
          More than one shared variable mandates synchronization
          usually, watch out!
Problem            Approach              Nature of fixes          War Stories



Lock granularity and lockless




          When you fix lock contention somewhere, ensure it hasn’t
          moved elsewhere, making overall performance worse.
          More than one shared variable mandates synchronization
          usually, watch out!
          Atomic operations and globaly unique objects(like interned
          symbols) are your friends.
Problem         Approach                Nature of fixes   War Stories



Remember...




              Jumps places, unlike functional bugs

                    Look beyond local impact of a fix
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach               Nature of fixes       War Stories



Garbage Collection

          When you hear unresponsiveness is intermittent, be
          alarmed!
              It probably is GC
              It is likely not easy to fix
              Root cause may be you or a library
Problem            Approach               Nature of fixes       War Stories



Garbage Collection

          When you hear unresponsiveness is intermittent, be
          alarmed!
              It probably is GC
              It is likely not easy to fix
              Root cause may be you or a library
          Scan GC logs
Problem             Approach                   Nature of fixes               War Stories



Garbage Collection

          When you hear unresponsiveness is intermittent, be
          alarmed!
              It probably is GC
              It is likely not easy to fix
              Root cause may be you or a library
          Scan GC logs
          Profile to see if majority of allocations are from just a
          handful of calls
              If its several calls with each contributing very little to the pie,
              don’t even try fixing it.
Problem             Approach                   Nature of fixes               War Stories



Garbage Collection

          When you hear unresponsiveness is intermittent, be
          alarmed!
              It probably is GC
              It is likely not easy to fix
              Root cause may be you or a library
          Scan GC logs
          Profile to see if majority of allocations are from just a
          handful of calls
              If its several calls with each contributing very little to the pie,
              don’t even try fixing it.
          Understand GC ergonomics and each algorithm(if you
          have alternatives) in detail.
          Tweek your max/min permitted allocation judiciously,
          malloc/free are expensive system calls.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach                Nature of fixes            War Stories



SQL Database



          Optimize connection pool size.
              Expand is very expensive.
              Is dispatched on caller thread(unless your connection pool
              is super-human).
Problem            Approach                Nature of fixes            War Stories



SQL Database



          Optimize connection pool size.
              Expand is very expensive.
              Is dispatched on caller thread(unless your connection pool
              is super-human).
          Watch out for slow queries, it can snowball.
Problem            Approach                Nature of fixes            War Stories



SQL Database



          Optimize connection pool size.
              Expand is very expensive.
              Is dispatched on caller thread(unless your connection pool
              is super-human).
          Watch out for slow queries, it can snowball.
          Understand your DB settings, MVCC, table-locking
          scenarios, row-locking scenarios, transaction isolation
          tradeoffs.
Problem              Approach              Nature of fixes            War Stories



SQL Database



          Optimize connection pool size.
              Expand is very expensive.
              Is dispatched on caller thread(unless your connection pool
              is super-human).
          Watch out for slow queries, it can snowball.
          Understand your DB settings, MVCC, table-locking
          scenarios, row-locking scenarios, transaction isolation
          tradeoffs.
          Contd...
Problem            Approach              Nature of fixes   War Stories



SQL Database contd...


          Explain and Analyze are your friends.
Problem            Approach                 Nature of fixes       War Stories



SQL Database contd...


          Explain and Analyze are your friends.
          Understanding implications of file-system IO helps realize
          table scans can be horrible
              DB runs with fixed page cache.
              Table scan can wipe it all cold.
              Use indexes to avoid table scan.
              Indexes are order sensitive.
              Tweek inner queries to reduce selectivity.
Problem            Approach                 Nature of fixes       War Stories



SQL Database contd...


          Explain and Analyze are your friends.
          Understanding implications of file-system IO helps realize
          table scans can be horrible
              DB runs with fixed page cache.
              Table scan can wipe it all cold.
              Use indexes to avoid table scan.
              Indexes are order sensitive.
              Tweek inner queries to reduce selectivity.
          Every DB may be subtly different(joins can be more
          expensive than materialzed views(if you have em))
Problem            Approach                 Nature of fixes       War Stories



SQL Database contd...


          Explain and Analyze are your friends.
          Understanding implications of file-system IO helps realize
          table scans can be horrible
              DB runs with fixed page cache.
              Table scan can wipe it all cold.
              Use indexes to avoid table scan.
              Indexes are order sensitive.
              Tweek inner queries to reduce selectivity.
          Every DB may be subtly different(joins can be more
          expensive than materialzed views(if you have em))
          Denormalization can help crack tough nuts.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach              Nature of fixes            War Stories



Context Switch Thrashing

          Too few cores, too many processes will obviously hurt.
Problem            Approach              Nature of fixes            War Stories



Context Switch Thrashing

          Too few cores, too many processes will obviously hurt.
          Too many locks will cause too many sleeps, hence too
          much ctx switching.
Problem            Approach              Nature of fixes            War Stories



Context Switch Thrashing

          Too few cores, too many processes will obviously hurt.
          Too many locks will cause too many sleeps, hence too
          much ctx switching.
          Ctx switch may schedule a different process, causing CPU
          caches and TLB to get cold.
Problem            Approach               Nature of fixes            War Stories



Context Switch Thrashing

          Too few cores, too many processes will obviously hurt.
          Too many locks will cause too many sleeps, hence too
          much ctx switching.
          Ctx switch may schedule a different process, causing CPU
          caches and TLB to get cold.
          Too frequent IO will cause io-wait, hence switch.
              Use of swap can completely kill performance. Swap is for
              emergency, not for regular usage.
              Over logging can hurt, be careful!
              Concept of buffered IO exists for a reason, use it!
              Use memory mapped IO if it makes sense.
Problem            Approach               Nature of fixes            War Stories



Context Switch Thrashing

          Too few cores, too many processes will obviously hurt.
          Too many locks will cause too many sleeps, hence too
          much ctx switching.
          Ctx switch may schedule a different process, causing CPU
          caches and TLB to get cold.
          Too frequent IO will cause io-wait, hence switch.
              Use of swap can completely kill performance. Swap is for
              emergency, not for regular usage.
              Over logging can hurt, be careful!
              Concept of buffered IO exists for a reason, use it!
              Use memory mapped IO if it makes sense.
          Assigning very low max memory limit can cause frequent
          GCs, allocate enough memory so its your functional thread
          that works and not GC threads.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem               Approach             Nature of fixes      War Stories



Common Pitfalls

          Prejudice
              SSL is bad
              Using asynchronicity to avoid fixing root cause
Problem               Approach             Nature of fixes            War Stories



Common Pitfalls

          Prejudice
              SSL is bad
              Using asynchronicity to avoid fixing root cause
          Lack of testing: In Go context, myth of 150 agent limit.
Problem               Approach             Nature of fixes            War Stories



Common Pitfalls

          Prejudice
              SSL is bad
              Using asynchronicity to avoid fixing root cause
          Lack of testing: In Go context, myth of 150 agent limit.
          Lack of setup understanding: Misconception that 64 bit
          JVM is always bad.
Problem               Approach             Nature of fixes            War Stories



Common Pitfalls

          Prejudice
              SSL is bad
              Using asynchronicity to avoid fixing root cause
          Lack of testing: In Go context, myth of 150 agent limit.
          Lack of setup understanding: Misconception that 64 bit
          JVM is always bad.
          Atomic operations and globaly unique objects(like interned
          symbols) are your friends.
Problem               Approach             Nature of fixes              War Stories



Common Pitfalls

          Prejudice
              SSL is bad
              Using asynchronicity to avoid fixing root cause
          Lack of testing: In Go context, myth of 150 agent limit.
          Lack of setup understanding: Misconception that 64 bit
          JVM is always bad.
          Atomic operations and globaly unique objects(like interned
          symbols) are your friends.
          Unchartred waters
              ERB garbage churn
              Strange JVM allocations
              Single threaded database
              JRuby class hirarchy modification(runtime level) locks.
Problem        Approach             Nature of fixes   War Stories



Know what you are getting into...




                Know what you are getting into...
Problem             Approach               Nature of fixes            War Stories



Gist, to reiterate...

          Do not go overboard, know your end goal.
          Intution tends to work very poorly. Belive proof, not gut feel.
          Understand your environment. Know what to measure.
          Round pegs for round holes. Use appropriate tools.
          Jumps places, unlike functional bugs, look beyond local
          impact.
          Requires mechanical approach, attack worst problem first.
          Prefer correctness over performance.
          Reasoning every fix (of paramount importance in
          multithreaded app).
          Do one fix at a time, allows easy reasoning and
          verification, can be reverted.
          Don’t jump in the lake without knowing whats waiting
          inside.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem           Approach       Nature of fixes   War Stories



Garbage churn issues



          Debug log statements
Problem           Approach       Nature of fixes   War Stories



Garbage churn issues



          Debug log statements
          ERB rendering
Problem            Approach              Nature of fixes      War Stories



Garbage churn issues



          Debug log statements
          ERB rendering
          Pooling objects with significant memory footprint
              XML parser factory
              Buffered reader
Problem            Approach              Nature of fixes      War Stories



Garbage churn issues



          Debug log statements
          ERB rendering
          Pooling objects with significant memory footprint
              XML parser factory
              Buffered reader
          Stream being dereferenced into a string
Problem            Approach              Nature of fixes          War Stories



Garbage churn issues



          Debug log statements
          ERB rendering
          Pooling objects with significant memory footprint
              XML parser factory
              Buffered reader
          Stream being dereferenced into a string
          Thread locals to the rescue(used for caching big reusable
          objects)
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach              Nature of fixes           War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
Problem            Approach              Nature of fixes           War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
          Death of synchronized methods
Problem            Approach              Nature of fixes           War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
          Death of synchronized methods
          Copy on to stack(using lock), mutate locklessly, copy back
          to shared memory(using lock)
Problem            Approach              Nature of fixes           War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
          Death of synchronized methods
          Copy on to stack(using lock), mutate locklessly, copy back
          to shared memory(using lock)
          Use read/write lock when appropriate
Problem            Approach              Nature of fixes           War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
          Death of synchronized methods
          Copy on to stack(using lock), mutate locklessly, copy back
          to shared memory(using lock)
          Use read/write lock when appropriate
          Use atomic operations for lockess concurrency
Problem            Approach              Nature of fixes           War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
          Death of synchronized methods
          Copy on to stack(using lock), mutate locklessly, copy back
          to shared memory(using lock)
          Use read/write lock when appropriate
          Use atomic operations for lockess concurrency
          Concurrent data-structure != thread-safety
Problem            Approach              Nature of fixes             War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
          Death of synchronized methods
          Copy on to stack(using lock), mutate locklessly, copy back
          to shared memory(using lock)
          Use read/write lock when appropriate
          Use atomic operations for lockess concurrency
          Concurrent data-structure != thread-safety
          Take difficult design tradeoffs to keep code requiring a
          class of lock in one place
Problem            Approach              Nature of fixes             War Stories



Contention and reasoning issues

          Locking on interned strings generated for context sensitive
          locking
          Death of synchronized methods
          Copy on to stack(using lock), mutate locklessly, copy back
          to shared memory(using lock)
          Use read/write lock when appropriate
          Use atomic operations for lockess concurrency
          Concurrent data-structure != thread-safety
          Take difficult design tradeoffs to keep code requiring a
          class of lock in one place
          Double check locking can be very helpful in frequently
          called code
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach            Nature of fixes         War Stories



Caching


          Eager caching is dangerous(using miss => load => prime
          => hit works better in practice)
Problem            Approach               Nature of fixes           War Stories



Caching


          Eager caching is dangerous(using miss => load => prime
          => hit works better in practice)
          Testing cache with boundry conditions
              Test to ensure cache-hit
              Ensure boundry conditions are well understood, overflow is
              judiciously defined/used
Problem            Approach               Nature of fixes           War Stories



Caching


          Eager caching is dangerous(using miss => load => prime
          => hit works better in practice)
          Testing cache with boundry conditions
              Test to ensure cache-hit
              Ensure boundry conditions are well understood, overflow is
              judiciously defined/used
          LRU eviction is the right(only?) solution sometimes
Problem            Approach               Nature of fixes           War Stories



Caching


          Eager caching is dangerous(using miss => load => prime
          => hit works better in practice)
          Testing cache with boundry conditions
              Test to ensure cache-hit
              Ensure boundry conditions are well understood, overflow is
              judiciously defined/used
          LRU eviction is the right(only?) solution sometimes
          Double check cache invalidation flows, its hard, sometimes
          bleeds across classes.
Problem            Approach               Nature of fixes           War Stories



Caching


          Eager caching is dangerous(using miss => load => prime
          => hit works better in practice)
          Testing cache with boundry conditions
              Test to ensure cache-hit
              Ensure boundry conditions are well understood, overflow is
              judiciously defined/used
          LRU eviction is the right(only?) solution sometimes
          Double check cache invalidation flows, its hard, sometimes
          bleeds across classes.
          Be vigilant when using hirarchical caches(no
          associative-arrays or maps should be cached), its easy to
          run memory out with this.
Problem            Approach             Nature of fixes         War Stories



Caching contd...


          Do not cache uncommited data or invalidate cache before
          commit(transactional caches may come in handy for this).
Problem            Approach              Nature of fixes           War Stories



Caching contd...


          Do not cache uncommited data or invalidate cache before
          commit(transactional caches may come in handy for this).
          Ensure transactions use cached data iff no other
          transaction modifies it or ensure transactionally consistent
          invalidation.
Problem            Approach              Nature of fixes           War Stories



Caching contd...


          Do not cache uncommited data or invalidate cache before
          commit(transactional caches may come in handy for this).
          Ensure transactions use cached data iff no other
          transaction modifies it or ensure transactionally consistent
          invalidation.
          Understand your caching library, too low overflow limit may
          cause IO issues.
Problem            Approach              Nature of fixes           War Stories



Caching contd...


          Do not cache uncommited data or invalidate cache before
          commit(transactional caches may come in handy for this).
          Ensure transactions use cached data iff no other
          transaction modifies it or ensure transactionally consistent
          invalidation.
          Understand your caching library, too low overflow limit may
          cause IO issues.
          Use one and only one cache instance(its easy to end up in
          a memory runout with this). We had view fragments, urls,
          domain objects and view models cached in the same
          instance.
Problem              Approach      Nature of fixes   War Stories



Outline
      1   Problem
             Identify scope
             Acceptance criteria
      2   Approach
             How to analyze it
             Measure
             Fix
      3   Nature of fixes
             Synchronization
             GC
             DB
             Thrashing
             Pitfalls
      4   War Stories
             GC
             Locking
             Caching
Problem            Approach               Nature of fixes   War Stories



Javascript

          JS dom frequent offset/size calls hurt
Problem            Approach               Nature of fixes       War Stories



Javascript

          JS dom frequent offset/size calls hurt
          Framework provided foreach loops are bad, use native for
          loops.
Problem            Approach               Nature of fixes         War Stories



Javascript

          JS dom frequent offset/size calls hurt
          Framework provided foreach loops are bad, use native for
          loops.
          Using beautiful framework provided methods like filter, find
          and select can have detrimental consequences.
Problem            Approach               Nature of fixes         War Stories



Javascript

          JS dom frequent offset/size calls hurt
          Framework provided foreach loops are bad, use native for
          loops.
          Using beautiful framework provided methods like filter, find
          and select can have detrimental consequences.
          Avoid multiple lookups for the same element, use a loaded
          copy as far as possible.
Problem            Approach               Nature of fixes         War Stories



Javascript

          JS dom frequent offset/size calls hurt
          Framework provided foreach loops are bad, use native for
          loops.
          Using beautiful framework provided methods like filter, find
          and select can have detrimental consequences.
          Avoid multiple lookups for the same element, use a loaded
          copy as far as possible.
          Avoid CSS selectors in frequently called code.
Problem            Approach               Nature of fixes         War Stories



Javascript

          JS dom frequent offset/size calls hurt
          Framework provided foreach loops are bad, use native for
          loops.
          Using beautiful framework provided methods like filter, find
          and select can have detrimental consequences.
          Avoid multiple lookups for the same element, use a loaded
          copy as far as possible.
          Avoid CSS selectors in frequently called code.
          Avoid too much DOM churn.
Problem            Approach               Nature of fixes          War Stories



Javascript

          JS dom frequent offset/size calls hurt
          Framework provided foreach loops are bad, use native for
          loops.
          Using beautiful framework provided methods like filter, find
          and select can have detrimental consequences.
          Avoid multiple lookups for the same element, use a loaded
          copy as far as possible.
          Avoid CSS selectors in frequently called code.
          Avoid too much DOM churn.
          Use tools like Chromium dev toolset to nail leaks and
          profile methods.
Problem            Approach               Nature of fixes          War Stories



Javascript

          JS dom frequent offset/size calls hurt
          Framework provided foreach loops are bad, use native for
          loops.
          Using beautiful framework provided methods like filter, find
          and select can have detrimental consequences.
          Avoid multiple lookups for the same element, use a loaded
          copy as far as possible.
          Avoid CSS selectors in frequently called code.
          Avoid too much DOM churn.
          Use tools like Chromium dev toolset to nail leaks and
          profile methods.
          Auto-refreshing dashboards should refresh only visible and
          changed area.
Problem      Approach         Nature of fixes   War Stories



Questions?




                        Questions?

More Related Content

Viewers also liked

Pre production power point presentation
Pre production power point presentationPre production power point presentation
Pre production power point presentation
ecsmedia
 
Nivki 2010 - Oleg
Nivki 2010 - OlegNivki 2010 - Oleg
Nivki 2010 - Oleg
nivki
 
What is Mootup?
What is Mootup?What is Mootup?
What is Mootup?
Mootup
 
Pre production 1
Pre production 1Pre production 1
Pre production 1
ecsmedia
 
El desarrollo del pesamiento creativo en estudiantes de ingneireia
El desarrollo del pesamiento creativo en estudiantes de ingneireiaEl desarrollo del pesamiento creativo en estudiantes de ingneireia
El desarrollo del pesamiento creativo en estudiantes de ingneireia
crojas6
 
Eurosocialnetwork THE PROJECT
Eurosocialnetwork THE PROJECTEurosocialnetwork THE PROJECT
Eurosocialnetwork THE PROJECT
Irecoop Toscana
 
Channel four ethos presentation
Channel four ethos presentationChannel four ethos presentation
Channel four ethos presentation
beealex
 
Research text and music
Research text and musicResearch text and music
Research text and music
beealex
 

Viewers also liked (18)

Sea
SeaSea
Sea
 
Pre production power point presentation
Pre production power point presentationPre production power point presentation
Pre production power point presentation
 
Nivki 2010 - Oleg
Nivki 2010 - OlegNivki 2010 - Oleg
Nivki 2010 - Oleg
 
What is Mootup?
What is Mootup?What is Mootup?
What is Mootup?
 
Geração de jesus - Jotta A
Geração de jesus - Jotta AGeração de jesus - Jotta A
Geração de jesus - Jotta A
 
2010 year end tax planning
2010 year end tax planning2010 year end tax planning
2010 year end tax planning
 
upload test2
upload test2upload test2
upload test2
 
El circo de los cuentos
El circo de los cuentosEl circo de los cuentos
El circo de los cuentos
 
Pre production 1
Pre production 1Pre production 1
Pre production 1
 
El desarrollo del pesamiento creativo en estudiantes de ingneireia
El desarrollo del pesamiento creativo en estudiantes de ingneireiaEl desarrollo del pesamiento creativo en estudiantes de ingneireia
El desarrollo del pesamiento creativo en estudiantes de ingneireia
 
Eurosocialnetwork THE PROJECT
Eurosocialnetwork THE PROJECTEurosocialnetwork THE PROJECT
Eurosocialnetwork THE PROJECT
 
Test
TestTest
Test
 
Channel four ethos presentation
Channel four ethos presentationChannel four ethos presentation
Channel four ethos presentation
 
Emerging Themes in Asian Prepaid Regulation
Emerging Themes in Asian Prepaid RegulationEmerging Themes in Asian Prepaid Regulation
Emerging Themes in Asian Prepaid Regulation
 
Research text and music
Research text and musicResearch text and music
Research text and music
 
1 Інвестиційний потенціал області - Микола Джига
1 Інвестиційний потенціал області - Микола Джига1 Інвестиційний потенціал області - Микола Джига
1 Інвестиційний потенціал області - Микола Джига
 
Компьютерные курсы для пожилых людей. Занятия 1 4
Компьютерные курсы для пожилых людей. Занятия 1 4Компьютерные курсы для пожилых людей. Занятия 1 4
Компьютерные курсы для пожилых людей. Занятия 1 4
 
libridjets-NTNU prosjekt, e-reader, students working with new technology
libridjets-NTNU prosjekt, e-reader, students working with new technologylibridjets-NTNU prosjekt, e-reader, students working with new technology
libridjets-NTNU prosjekt, e-reader, students working with new technology
 

Similar to Performance Tuning 101 (6)

Phantom Doctrine: a Design Post-Mortem
Phantom Doctrine: a Design Post-MortemPhantom Doctrine: a Design Post-Mortem
Phantom Doctrine: a Design Post-Mortem
 
Climbing Rocks & Coding Blocks
Climbing Rocks  & Coding BlocksClimbing Rocks  & Coding Blocks
Climbing Rocks & Coding Blocks
 
Discount Usability Testing for Agile Teams
Discount Usability Testing for Agile TeamsDiscount Usability Testing for Agile Teams
Discount Usability Testing for Agile Teams
 
An Introduction into Creative Thinking
An Introduction into Creative ThinkingAn Introduction into Creative Thinking
An Introduction into Creative Thinking
 
Outsandingpresentations bzcmp-v2-100529150452-phpapp01
Outsandingpresentations bzcmp-v2-100529150452-phpapp01Outsandingpresentations bzcmp-v2-100529150452-phpapp01
Outsandingpresentations bzcmp-v2-100529150452-phpapp01
 
Adventure pro forma.pptx
Adventure pro forma.pptxAdventure pro forma.pptx
Adventure pro forma.pptx
 

Recently uploaded

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
Muhammad Subhan
 

Recently uploaded (20)

Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties ReimaginedEasier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 

Performance Tuning 101

  • 1. Problem Approach Nature of fixes War Stories Performance Tuning 101 An introduction to performance tuning an application in managed environment Pavan KS1 Janmejay Singh2 1 mail: itspanzi@gmail.com blog: http://itspanzi.blogspot.com 2 mail: singh.janmejay@gmail.com blog: http://codehunk.wordpress.com Rootconf Bangalore, May 2012
  • 2. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems.
  • 3. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal.
  • 4. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel.
  • 5. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure.
  • 6. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools.
  • 7. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact.
  • 8. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first.
  • 9. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance.
  • 10. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app).
  • 11. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted.
  • 12. Problem Approach Nature of fixes War Stories Gist Don’t be out there with a gun looking for problems. Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted. Don’t jump in the lake without knowing whats waiting inside.
  • 13. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite)
  • 14. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp.
  • 15. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment.
  • 16. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment. GNU/Linux
  • 17. Problem Approach Nature of fixes War Stories Context App in reference: ThoughtWorks Studios - Go (CI/CD suite) Java (backend) and JRuby on Rails (frontend) webapp. Managed environment. GNU/Linux Approach and learning is independent of tech-stack.
  • 18. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 19. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 20. Problem Approach Nature of fixes War Stories Don’t go looking for performance issues Doing it proactively is a wild goose chase! Varying parameters and running into a real problem is improbable, if not impossible.
  • 21. Problem Approach Nature of fixes War Stories Remember... Don’t be out there with a gun looking for problems. Its easier to reproduce and fix a performance issue when you have seen it happen in a real setup. That way, you also know for sure, it is real.
  • 22. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow
  • 23. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow
  • 24. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule
  • 25. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late
  • 26. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late SCM changes reflect after a long time
  • 27. Problem Approach Nature of fixes War Stories My app is slow, it doesn’t respond... In case of Go, this could mean, The dashboard is slow Javascript is slow Builds taking too long to schedule Worker processes picking up work too late SCM changes reflect after a long time Process is hogging resources(requires 16 cores, 8 gig ram for a basic setup)
  • 28. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 29. Problem Approach Nature of fixes War Stories Benchmark How slow is slow?
  • 30. Problem Approach Nature of fixes War Stories Benchmark How slow is slow? How fast is fast?
  • 31. Problem Approach Nature of fixes War Stories Benchmark How slow is slow? How fast is fast? Benchmark using realistic setup.
  • 32. Problem Approach Nature of fixes War Stories Benchmark How slow is slow? How fast is fast? Benchmark using realistic setup. Use a repeatable and automated setup.
  • 33. Problem Approach Nature of fixes War Stories Remember... Do not go overboard. Know your end goal.
  • 34. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 35. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix?
  • 36. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix? What if I stop at fix?
  • 37. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix? What if I stop at fix? Why do I need to iterate?
  • 38. Problem Approach Nature of fixes War Stories Trust in testing What if I start at fix? What if I stop at fix? Why do I need to iterate? How long should I iterate?
  • 39. Problem Approach Nature of fixes War Stories Remember... Intution tends to work very poorly. Belive proof, not gut feel.
  • 40. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 41. Problem Approach Nature of fixes War Stories What to measure Easy and obvious: Page response time
  • 42. Problem Approach Nature of fixes War Stories What to measure Easy and obvious: Page response time Not so easy: App throughput Enqueue/Dequeue time for message queues CPU/IO churn Database performance Lock contention Managed environment specific parameters(Memory usage, GC churn etc). Environmental issues(too many processes, too low memory, very slow IO).
  • 43. Problem Approach Nature of fixes War Stories Remember... Understand your environment. Know what to measure.
  • 44. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf
  • 45. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts
  • 46. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics.
  • 47. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify.
  • 48. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler.
  • 49. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler.
  • 50. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler. Managed env specific params: GC logs, GC profiling.
  • 51. Problem Approach Nature of fixes War Stories Tools to measure User load ab htperf App throughput: measure app-level churn(creation/processing/deletion/delivery) counts Enqueue/Dequeue time: Use broker’s management console/logs/statistics. CPU/IO churn: Load Avg, iostat, klogd, inotify. Database performance: Profiler. Lock contention: Profiler. Managed env specific params: GC logs, GC profiling. Environmental issues: ps, dmesg, /proc/<pid>/status, /proc/net/dev, syslog, messages.
  • 52. Problem Approach Nature of fixes War Stories Remember... Round pegs for round holes. Use appropriate tools. Configure them correctly with least possible overhead.
  • 53. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 54. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code.
  • 55. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical.
  • 56. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem.
  • 57. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix.
  • 58. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous.
  • 59. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads.
  • 60. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads. Attack one and only one problem.
  • 61. Problem Approach Nature of fixes War Stories How to carry out fixes Understand libraries: read fine print in documentation, or better yet, read code. Be mechanical. Use exhaggerated load to maginify the problem. Use the same load profile to test your fix. Be rigorous. Mob pair on complicated fixes, especially fixes involving data mutation across threads. Attack one and only one problem. Having said all that, correctness always comes first.
  • 62. Problem Approach Nature of fixes War Stories Remember... Requires mechanical approach Attack worst problem first Reason every fix Reasoning is of paramount importance in multithreaded app Do one fix at a time Allows easy reasoning and verification, can be reverted Prefer correctness over performance
  • 63. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 64. Problem Approach Nature of fixes War Stories Lock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse.
  • 65. Problem Approach Nature of fixes War Stories Lock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse. More than one shared variable mandates synchronization usually, watch out!
  • 66. Problem Approach Nature of fixes War Stories Lock granularity and lockless When you fix lock contention somewhere, ensure it hasn’t moved elsewhere, making overall performance worse. More than one shared variable mandates synchronization usually, watch out! Atomic operations and globaly unique objects(like interned symbols) are your friends.
  • 67. Problem Approach Nature of fixes War Stories Remember... Jumps places, unlike functional bugs Look beyond local impact of a fix
  • 68. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 69. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library
  • 70. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs
  • 71. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs Profile to see if majority of allocations are from just a handful of calls If its several calls with each contributing very little to the pie, don’t even try fixing it.
  • 72. Problem Approach Nature of fixes War Stories Garbage Collection When you hear unresponsiveness is intermittent, be alarmed! It probably is GC It is likely not easy to fix Root cause may be you or a library Scan GC logs Profile to see if majority of allocations are from just a handful of calls If its several calls with each contributing very little to the pie, don’t even try fixing it. Understand GC ergonomics and each algorithm(if you have alternatives) in detail. Tweek your max/min permitted allocation judiciously, malloc/free are expensive system calls.
  • 73. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 74. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human).
  • 75. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball.
  • 76. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball. Understand your DB settings, MVCC, table-locking scenarios, row-locking scenarios, transaction isolation tradeoffs.
  • 77. Problem Approach Nature of fixes War Stories SQL Database Optimize connection pool size. Expand is very expensive. Is dispatched on caller thread(unless your connection pool is super-human). Watch out for slow queries, it can snowball. Understand your DB settings, MVCC, table-locking scenarios, row-locking scenarios, transaction isolation tradeoffs. Contd...
  • 78. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends.
  • 79. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity.
  • 80. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity. Every DB may be subtly different(joins can be more expensive than materialzed views(if you have em))
  • 81. Problem Approach Nature of fixes War Stories SQL Database contd... Explain and Analyze are your friends. Understanding implications of file-system IO helps realize table scans can be horrible DB runs with fixed page cache. Table scan can wipe it all cold. Use indexes to avoid table scan. Indexes are order sensitive. Tweek inner queries to reduce selectivity. Every DB may be subtly different(joins can be more expensive than materialzed views(if you have em)) Denormalization can help crack tough nuts.
  • 82. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 83. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt.
  • 84. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching.
  • 85. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold.
  • 86. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold. Too frequent IO will cause io-wait, hence switch. Use of swap can completely kill performance. Swap is for emergency, not for regular usage. Over logging can hurt, be careful! Concept of buffered IO exists for a reason, use it! Use memory mapped IO if it makes sense.
  • 87. Problem Approach Nature of fixes War Stories Context Switch Thrashing Too few cores, too many processes will obviously hurt. Too many locks will cause too many sleeps, hence too much ctx switching. Ctx switch may schedule a different process, causing CPU caches and TLB to get cold. Too frequent IO will cause io-wait, hence switch. Use of swap can completely kill performance. Swap is for emergency, not for regular usage. Over logging can hurt, be careful! Concept of buffered IO exists for a reason, use it! Use memory mapped IO if it makes sense. Assigning very low max memory limit can cause frequent GCs, allocate enough memory so its your functional thread that works and not GC threads.
  • 88. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 89. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause
  • 90. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit.
  • 91. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad.
  • 92. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad. Atomic operations and globaly unique objects(like interned symbols) are your friends.
  • 93. Problem Approach Nature of fixes War Stories Common Pitfalls Prejudice SSL is bad Using asynchronicity to avoid fixing root cause Lack of testing: In Go context, myth of 150 agent limit. Lack of setup understanding: Misconception that 64 bit JVM is always bad. Atomic operations and globaly unique objects(like interned symbols) are your friends. Unchartred waters ERB garbage churn Strange JVM allocations Single threaded database JRuby class hirarchy modification(runtime level) locks.
  • 94. Problem Approach Nature of fixes War Stories Know what you are getting into... Know what you are getting into...
  • 95. Problem Approach Nature of fixes War Stories Gist, to reiterate... Do not go overboard, know your end goal. Intution tends to work very poorly. Belive proof, not gut feel. Understand your environment. Know what to measure. Round pegs for round holes. Use appropriate tools. Jumps places, unlike functional bugs, look beyond local impact. Requires mechanical approach, attack worst problem first. Prefer correctness over performance. Reasoning every fix (of paramount importance in multithreaded app). Do one fix at a time, allows easy reasoning and verification, can be reverted. Don’t jump in the lake without knowing whats waiting inside.
  • 96. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 97. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements
  • 98. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering
  • 99. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader
  • 100. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader Stream being dereferenced into a string
  • 101. Problem Approach Nature of fixes War Stories Garbage churn issues Debug log statements ERB rendering Pooling objects with significant memory footprint XML parser factory Buffered reader Stream being dereferenced into a string Thread locals to the rescue(used for caching big reusable objects)
  • 102. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 103. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking
  • 104. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods
  • 105. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock)
  • 106. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate
  • 107. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency
  • 108. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety
  • 109. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety Take difficult design tradeoffs to keep code requiring a class of lock in one place
  • 110. Problem Approach Nature of fixes War Stories Contention and reasoning issues Locking on interned strings generated for context sensitive locking Death of synchronized methods Copy on to stack(using lock), mutate locklessly, copy back to shared memory(using lock) Use read/write lock when appropriate Use atomic operations for lockess concurrency Concurrent data-structure != thread-safety Take difficult design tradeoffs to keep code requiring a class of lock in one place Double check locking can be very helpful in frequently called code
  • 111. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 112. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice)
  • 113. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used
  • 114. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes
  • 115. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes Double check cache invalidation flows, its hard, sometimes bleeds across classes.
  • 116. Problem Approach Nature of fixes War Stories Caching Eager caching is dangerous(using miss => load => prime => hit works better in practice) Testing cache with boundry conditions Test to ensure cache-hit Ensure boundry conditions are well understood, overflow is judiciously defined/used LRU eviction is the right(only?) solution sometimes Double check cache invalidation flows, its hard, sometimes bleeds across classes. Be vigilant when using hirarchical caches(no associative-arrays or maps should be cached), its easy to run memory out with this.
  • 117. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this).
  • 118. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation.
  • 119. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation. Understand your caching library, too low overflow limit may cause IO issues.
  • 120. Problem Approach Nature of fixes War Stories Caching contd... Do not cache uncommited data or invalidate cache before commit(transactional caches may come in handy for this). Ensure transactions use cached data iff no other transaction modifies it or ensure transactionally consistent invalidation. Understand your caching library, too low overflow limit may cause IO issues. Use one and only one cache instance(its easy to end up in a memory runout with this). We had view fragments, urls, domain objects and view models cached in the same instance.
  • 121. Problem Approach Nature of fixes War Stories Outline 1 Problem Identify scope Acceptance criteria 2 Approach How to analyze it Measure Fix 3 Nature of fixes Synchronization GC DB Thrashing Pitfalls 4 War Stories GC Locking Caching
  • 122. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt
  • 123. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops.
  • 124. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences.
  • 125. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible.
  • 126. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code.
  • 127. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn.
  • 128. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn. Use tools like Chromium dev toolset to nail leaks and profile methods.
  • 129. Problem Approach Nature of fixes War Stories Javascript JS dom frequent offset/size calls hurt Framework provided foreach loops are bad, use native for loops. Using beautiful framework provided methods like filter, find and select can have detrimental consequences. Avoid multiple lookups for the same element, use a loaded copy as far as possible. Avoid CSS selectors in frequently called code. Avoid too much DOM churn. Use tools like Chromium dev toolset to nail leaks and profile methods. Auto-refreshing dashboards should refresh only visible and changed area.
  • 130. Problem Approach Nature of fixes War Stories Questions? Questions?