This document discusses performance metrics for monitoring and optimizing a social network application. It recommends collecting metrics at various levels, including development, system health, and page-level performance. It describes tools for each level like Debug Toolbar, New Relic, Graphite, Cloudwatch, and PgFouine. Graphite is highlighted as a tool for storing vast amounts of metrics data that can then be queried and visualized in different ways. The document provides examples of specific metrics that could be tracked, like database query times, cache performance, task processing times, and deploy monitoring. It emphasizes that collecting a wide range of metrics across the application stack helps identify bottlenecks and optimize performance.

1. Performance metrics
for a social network

2. About Me
• Thierry Schellenbach
• Founder/ CTO Fashiolista
• Author of Django Facebook
• Github/tschellenbach
• Blog: mellowmorning.com
• @tschellenbach

3. Global Fashion Discovery

7. 5.000.000+
8.000.000+

8. Growth
2nd largest fashion community
• 1mln members
• 17mln loves/month
• 94mln non-bot pageviews

9. Powered By
• Django/Python
• PostgreSQL
• Solr
• Redis
• Celery
• AWS/ Ubuntu
• Nginx/ Gunicorn/ Supervisor

10. Sexy Metrics driven optimization
Hard Because
• All content is personalized
• Activity is clustered around a few
users (>100k followers)
• Individual users are insanely
active (7 hours in a day is
normal)
• Social network, can’t easily shard
data

11. Speed is a Feature

12. Metrics across the board
• Development
– Spot things early on, wrong usage of ORM etc
• System Health
– Is my DB healthy, my Redis cluster etc
• Page level
– Why is my page slow
– What is the average speed of the components
(DB, Redis, Solr etc)

13. Tools we use
Development System Health Page Level
• Cloudwatch • New Relic
• Debug toolbar
• Munin
– Cache calls • Graphite
• Nagios
– Graphite
Timings • DB slow log
– Queries and • Redis slow log
their explains • Integration Tests
– Duplicate query • PgFouine
detection

14. Development
StatsD
Duplicates
Cache Calls

15. Today’s Presentation
New Relic Graphite PgFouine
• Dashboard, High • Stash all data, • Understand
level insights query it any way what keeps
you want your DB busy
• Tool, not a
dashboard

16. New Relic
• Frontend -> App ->
Components (DB, Solr, etc.)
• Breaks page performance
down into it’s components
• Tracks deploys and compares
before and after

17. Are you Supported?
• Ruby • Pip install newrelic
• Java • Edit the .ini
• .NET • Add the WSGI middleware
• PHP • Wait for Magic
• Python

18. End user load times
• Drill down all the way to Database calls
• The purple line is our app, the rest frontend
Frontend
(97%)
App

19. Global page loads

20. Page Level
• Average frontend performance per page
• Click to view App level breakdown
Page. Not URL. To App Level

21. Drill down/ App overview
History
Memcached
DB Query

22. Database
• See which tables are
under most load
• See which pages cause the
load
• Development over time

23. Deploys

24. Deploys part Two
Response Time Pre & Post
Memory Utilization

25. Background Task
Number of Task
calls (sample)

26. Graphite Insights
• NewRelic has the overview,
Graphite the detail
• Open Source!
• Throw data at it via UDP
• Popularized by Etsy
(see mellowmorning.com for
link)

27. It’s Complicated

28. Tracks Everything

29. Setup
• Track using StatsD
– Support for (PHP, Python, Ruby, Node, Java)
• Hierarchy (python example)
• get.<app>.<view>.<component>
with request.timings('get.user.profile_page.sql'):
print ‘database query here’

30. Data tool/ Not a dashboard
• Wildcards
– get.<app>.<view>.*.upper_90
– get.<app>.*.redis.zadd.upper_90
– limit(sortByMaxima(get.<app>.<view>.*.up
per_90),4)

31. /style/<user>/ performance
Memcached
Slowdown
ZADD
Set Many

32. Including Functional parts of Pages
• More like this part is
tracked
• Solr & Redis Cache

33. What we Track
• Loadtime per bit of functionality
• Database calls per DB
• 90th percentile load times
• Task broker roundtrip times
• Facebook API calls

34. PgFouine
• Run on samples of all queries (say 5m)
• Not just slow queries
• Repeating a simple query many times is also
wrong, PgFouine finds it
• See Instagram’s fabric snippet
• https://gist.github.com/2307647

35. PgFouine Continued
Queries that took up
the most time (N)
• Spots issues with
many small queries Normalized
Compare multiple
reports

36. PgFouine Tips
• My colleague wrote a fast C++ version
• github.com/WoLpH/pg_query_analyser
Also look at:
• Pg Stat Statement
• Pg Badger

37. Concluding
New Relic Graphite PgFouine
• Dashboard, High • Stash all data, • Understand
level insights query it any way what keeps
you want your DB busy
• Tool, not a
dashboard

38. Q&A
We’re Searching for Django Developers & Linux
system administrators!
Fashiolista.com/jobs
Open source projects:
Github.com/tschellenbach
Try Django Facebook!