Your SlideShare is downloading. ×
Avoiding Performance Issues: When and How to Debug Production
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Avoiding Performance Issues: When and How to Debug Production


Published on

Published in: Technology

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Avoiding  Performance  Issues:   When  and  How  to  Debug  Produc3on  
  • 2. Best  Prac6ces   Best Practice Steps to Take Assure code quality •  Version control •  Test coverage Dedicate time to performance and security •  Dedicated positions are ideal •  For new projects, build into time estimates •  For older projects, dedicated time to revisit performance/security Avoid technical debt •  Be proactive about updating versions •  Don’t write untested code Continuous integration 2 •  •  •  •  Avoid doing work on production! Proper dev/stage/prod workflow Use CI tools to integrate and test code commits Stage should be a (sanitized) clone of prod, but completely separated from prod •  Emphasize that doing work ‘near’ production is undesirable, but doing work on a clone of production is desirable
  • 3. But  …  Stuff  Happens   Unexpected Situations •  Relevant employees are inaccessible •  Unknown dependencies •  Emergencies affecting the stakeholder Bugs that only show up at scale •  Table locks •  Queries lacking indexes or requiring temp tables to disk (occurs with blob/ text columns, string cols >512 char in group by, distinct or unions) •  Users with atypical behavior, like posting thousands of comments, dozens of “tabs” 3 Bugs caused by third parties •  ‘Noisy neighbor’ problems •  Rolling outages •  DNS
  • 4. 5  Steps  to…   Wri6ng  code  that  won't   cause  debugging  headaches   Structuring  deployments  to  keep   them  responsive  and  reversible   Con6nuous  monitoring  across  produc6on   Agility  in  following  problems  through  complex   systems   Making  the  right  team  members  are  aware  of  problems   4
  • 5. Step  1:  Wri3ng  code  that  won't  cause  debugging  headaches   1 2 Dynamic dispatch 3 Evented or highly asynchronous code 4 Difficult data structures (e.g., closures) 5 5 Confusing execution flow Include monitoring considerations in your codebase
  • 6. Step  2:  Structuring  deployments  to  keep  them  responsive  &  reversible   1 2 Consider the cost of backing out: having to do a full database restore can take a huge amount of time on a large site, so you ideally want to be able to do selective reverts 3 Know what changes are irreversible and what aren’t 4 Keep deployments as small as possible 5 Gradually roll out production changes, rather than all at once 6 Consider where you can make architectural decisions that enable fast changes, e.g. setting a very low DNS TTL 7 6 Knowledge of what goes into a deployment Deploy during office hours when possible, and not on Fridays
  • 7. Step  3:  Con3nuous  monitoring  across  produc3on   1 2 Log files 3 Availability 4 Latency 5 7 Infrastructure health Cache effectiveness
  • 8. Step  4:  Agility  in  following  problems  through  complex  systems   1 2 Might require dedicated time for building tooling 3 Playbooks for when it goes well, and also for when it doesn’t 4 8 A big part of why DevOps matters Making monitoring accessible
  • 9. Step  5:  Making  the  right  team  members  are  aware  of  problems   Doesn’t  have  to  be  a  “flat   hierarchy”,  but  how  effec6ve  are   your  channels?   9
  • 10. Ques6ons?