Your SlideShare is downloading. ×
0
Avoiding	
  Performance	
  Issues:	
  
When	
  and	
  How	
  to	
  Debug	
  Produc3on	
  
Best	
  Prac6ces	
  
Best Practice

Steps to Take

Assure code quality

•  Version control
•  Test coverage

Dedicate time...
But	
  …	
  Stuff	
  Happens	
  

Unexpected Situations
•  Relevant employees are
inaccessible
•  Unknown dependencies
•  E...
5	
  Steps	
  to…	
  

Wri6ng	
  code	
  that	
  won't	
  
cause	
  debugging	
  headaches	
  
Structuring	
  deployments	...
Step	
  1:	
  Wri3ng	
  code	
  that	
  won't	
  cause	
  debugging	
  headaches	
  

1

2

Dynamic dispatch

3

Evented o...
Step	
  2:	
  Structuring	
  deployments	
  to	
  keep	
  them	
  responsive	
  &	
  reversible	
  
1

2

Consider the cos...
Step	
  3:	
  Con3nuous	
  monitoring	
  across	
  produc3on	
  

1

2

Log files

3

Availability

4

Latency

5

7

Infr...
Step	
  4:	
  Agility	
  in	
  following	
  problems	
  through	
  complex	
  systems	
  

1

2

Might require dedicated t...
Step	
  5:	
  Making	
  the	
  right	
  team	
  members	
  are	
  aware	
  of	
  problems	
  

Doesn’t	
  have	
  to	
  be...
Ques6ons?	
  
Upcoming SlideShare
Loading in...5
×

Avoiding Performance Issues: When and How to Debug Production

587

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
587
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Avoiding Performance Issues: When and How to Debug Production"

  1. 1. Avoiding  Performance  Issues:   When  and  How  to  Debug  Produc3on  
  2. 2. Best  Prac6ces   Best Practice Steps to Take Assure code quality •  Version control •  Test coverage Dedicate time to performance and security •  Dedicated positions are ideal •  For new projects, build into time estimates •  For older projects, dedicated time to revisit performance/security Avoid technical debt •  Be proactive about updating versions •  Don’t write untested code Continuous integration 2 •  •  •  •  Avoid doing work on production! Proper dev/stage/prod workflow Use CI tools to integrate and test code commits Stage should be a (sanitized) clone of prod, but completely separated from prod •  Emphasize that doing work ‘near’ production is undesirable, but doing work on a clone of production is desirable
  3. 3. But  …  Stuff  Happens   Unexpected Situations •  Relevant employees are inaccessible •  Unknown dependencies •  Emergencies affecting the stakeholder Bugs that only show up at scale •  Table locks •  Queries lacking indexes or requiring temp tables to disk (occurs with blob/ text columns, string cols >512 char in group by, distinct or unions) •  Users with atypical behavior, like posting thousands of comments, dozens of “tabs” 3 Bugs caused by third parties •  ‘Noisy neighbor’ problems •  Rolling outages •  DNS
  4. 4. 5  Steps  to…   Wri6ng  code  that  won't   cause  debugging  headaches   Structuring  deployments  to  keep   them  responsive  and  reversible   Con6nuous  monitoring  across  produc6on   Agility  in  following  problems  through  complex   systems   Making  the  right  team  members  are  aware  of  problems   4
  5. 5. Step  1:  Wri3ng  code  that  won't  cause  debugging  headaches   1 2 Dynamic dispatch 3 Evented or highly asynchronous code 4 Difficult data structures (e.g., closures) 5 5 Confusing execution flow Include monitoring considerations in your codebase
  6. 6. Step  2:  Structuring  deployments  to  keep  them  responsive  &  reversible   1 2 Consider the cost of backing out: having to do a full database restore can take a huge amount of time on a large site, so you ideally want to be able to do selective reverts 3 Know what changes are irreversible and what aren’t 4 Keep deployments as small as possible 5 Gradually roll out production changes, rather than all at once 6 Consider where you can make architectural decisions that enable fast changes, e.g. setting a very low DNS TTL 7 6 Knowledge of what goes into a deployment Deploy during office hours when possible, and not on Fridays
  7. 7. Step  3:  Con3nuous  monitoring  across  produc3on   1 2 Log files 3 Availability 4 Latency 5 7 Infrastructure health Cache effectiveness
  8. 8. Step  4:  Agility  in  following  problems  through  complex  systems   1 2 Might require dedicated time for building tooling 3 Playbooks for when it goes well, and also for when it doesn’t 4 8 A big part of why DevOps matters Making monitoring accessible
  9. 9. Step  5:  Making  the  right  team  members  are  aware  of  problems   Doesn’t  have  to  be  a  “flat   hierarchy”,  but  how  effec6ve  are   your  channels?   9
  10. 10. Ques6ons?  
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×